---
title: "Document-Level Relation Extraction with Adaptive Focal Loss"
description: "The authors propose a semi-supervised framework with three parts: Axial attention: a pre-trained… 32 references, cited by 49 papers here."
authors: ["Qingyu Tan", "Ruidan He", "Lidong Bing", "Hwee Tou Ng"]
year: 2022
venue: "Findings of the Association for Computational Linguistics: ACL 2022"
source: "https://aclanthology.org/2022.findings-acl.132/"
doi: "10.18653/v1/2022.findings-acl.132"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tan-2022-document-level-relation-extraction-adaptive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tan-2022-document-level-relation-extraction-adaptive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation

# Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation

**Authors:** [Qingyu Tan](https://md.research.elamur.ai/free/en/authors/qingyu-tan/), [Ruidan He](https://md.research.elamur.ai/free/en/authors/ruidan-he/), [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) and [Hwee Tou Ng](https://md.research.elamur.ai/free/en/authors/hwee-tou-ng/) · **Year:** 2022 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2022](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2022](https://md.research.elamur.ai/free/en/venues/findings/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.findings-acl.132/) · [PDF](https://aclanthology.org/2022.findings-acl.132.pdf) · [DOI 10.18653/v1/2022.findings-acl.132](https://doi.org/10.18653/v1/2022.findings-acl.132) · [OpenAlex](https://openalex.org/W4226031465)

**Citations:** 120 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/tan-2022-document-level-relation-extraction-adaptive/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses document-level relation extraction (DocRE), where relations must be identified between every ordered pair of given entities across a whole document. Most pairs carry no relation, and the positive relation types follow a heavily skewed distribution. The authors propose a semi-supervised framework with three parts:

- **Axial attention:** a pre-trained language model encodes the entity-pair table, and axial attention lets each pair consult other pairs that share its subject or object. This targets two-hop reasoning.
- **Adaptive Focal Loss:** a new loss that gives rare, long-tail classes more weight in training.
- **Knowledge distillation:** a teacher trained on human-annotated data produces soft labels on distantly supervised documents. A student is pre-trained on these soft labels together with the distant labels, then fine-tuned on the annotated set.

On DocRED and HacRED the system consistently beats strong baselines. On the DocRED leaderboard it surpasses the previous best system. An error analysis suggests that many apparent false positives are true facts missing from the annotation. This matters for anyone building document-level information extraction resources and evaluation benchmarks.

## Key points
- On the DocRED leaderboard, the model (submitted as KD-Roberta) beats the earlier SOTA, SSAN-Adapt, by 1.36 F1 and 1.46 Ign_F1.
- DocRED has 96 relation types, and the 10 most frequent account for 59.4% of relation labels. This imbalance motivates the Adaptive Focal Loss.
- Error breakdown on the DocRED dev set: 51.4% correct, 23.7% extra predictions, 23.4% missed, and only 1.5% wrong relation type.
- Reducing the task to binary "related or not" raises dev F1 only slightly, from 67.12 to 68.64. The authors attribute part of the remaining error to incomplete gold annotations.
- Code and data are released at github.com/tonytan48/KD-DocRE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 28 works · 6,115 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 15 works · 3,209 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Qingyu Tan, Ruidan He, Lidong Bing and Hwee Tou Ng. 2022. Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation. Findings of the Association for Computational Linguistics: ACL 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tan2022document,
  title = {{Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation}},
  author = {Qingyu Tan, Ruidan He, Lidong Bing and Hwee Tou Ng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.132},
  url = {https://doi.org/10.18653/v1/2022.findings-acl.132}
}
```

- [← previous Generating Sequences by Learning to Self-Correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/)
- [next → Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://md.research.elamur.ai/free/en/papers/ramamurthy-2022-reinforcement-learning-natural-language-processing/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. [Matching the blanks: Distributional similarity for relation learning](https://md.research.elamur.ai/free/en/papers/soares-2019-matching-blanks-distributional-similarity-relation/). In Proceedings of ACL.
2. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. HacRED: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Qiao+Cheng%2C+Juntao+Liu%2C+Xiaoye+Qu%2C+Jin+Zhao%2C+Jiaqing+Liang%2C+Zhefeng+Wang%2C+Baoxing+Huai%2C+Nicholas+Jing+Yuan%2C+and+Yanghua+Xiao%2E+2021%2E+HacRED%3A+A+large%2Dscale+relation+extraction+dataset+toward+hard+cases+)
3. Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. [Connecting the dots: Document-level neural relation extraction with edge-oriented graphs](https://md.research.elamur.ai/free/en/papers/christopoulou-2019-connecting-dots-document-level-neural/). In Proceedings of EMNLP.
4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL.
6. Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Jun+Feng%2C+Minlie+Huang%2C+Li+Zhao%2C+Yang+Yang%2C+and+Xiaoyan+Zhu%2E+2018%2E+Reinforcement+learning+for+relation+classification+from+noisy+data%2E+In+Proceedings+of+AAAI%2E)
7. Jingye Li, Kang Xu, Fei Li, Hao Fei, Yafeng Ren, and Donghong Ji. 2021. Mrn: A locally and globally mention-based reasoning network for document-level relation extraction. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Jingye+Li%2C+Kang+Xu%2C+Fei+Li%2C+Hao+Fei%2C+Yafeng+Ren%2C+and+Donghong+Ji%2E+2021%2E+Mrn%3A+A+locally+and+globally+mention%2Dbased+reasoning+network+for+document%2Dlevel+relation+extraction%2E+In+Findings+of+ACL%2E)
8. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. [Focal loss for dense object detection](https://md.research.elamur.ai/free/en/papers/lin-2017-focal-loss-dense-object-detection/). In Proceedings of ICCV.
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
10. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+Proceedings+of+ICLR%2E)
11. Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. [Reasoning with latent structure refinement for document-level relation extraction](https://md.research.elamur.ai/free/en/papers/nan-2020-reasoning-latent-structure-refinement-document/). In Proceedings of ACL.
12. Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. [Learning from context or names? An empirical study on neural relation extraction](https://md.research.elamur.ai/free/en/papers/peng-2020-learning-context-or-names-empirical/). In Proceedings of EMNLP.
13. Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence nary relation extraction with graph lstms. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nanyun+Peng%2C+Hoifung+Poon%2C+Chris+Quirk%2C+Kristina+Toutanova%2C+and+Wen%2Dtau+Yih%2E+2017%2E+Cross%2Dsentence+nary+relation+extraction+with+graph+lstms%2E+Transactions+of+the+Association+for+Computational+Linguisti)
14. Pengda Qin, Weiran Xu, and William Yang Wang. 2018. DSGAN: Generative adversarial training for distant supervision relation extraction. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Pengda+Qin%2C+Weiran+Xu%2C+and+William+Yang+Wang%2E+2018%2E+DSGAN%3A+Generative+adversarial+training+for+distant+supervision+relation+extraction%2E+In+Proceedings+of+ACL%2E)
15. Chris Quirk and Hoifung Poon. 2017. [Distant supervision for relation extraction beyond the sentence boundary](https://md.research.elamur.ai/free/en/papers/quirk-2017-distant-supervision-relation-extraction-beyond/). In Proceedings of EACL.
16. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks. [find](https://scholar.google.com/scholar?q=Franco+Scarselli%2C+Marco+Gori%2C+Ah+Chung+Tsoi%2C+Markus+Hagenbuchner%2C+and+Gabriele+Monfardini%2E+2008%2E+The+graph+neural+network+model%2E+IEEE+Transactions+on+Neural+Networks%2E)
17. Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020. HIN: hierarchical inference network for document-level relation extraction. In Proceedings of KDD. [find](https://scholar.google.com/scholar?q=Hengzhu+Tang%2C+Yanan+Cao%2C+Zhenyu+Zhang%2C+Jiangxia+Cao%2C+Fang+Fang%2C+Shi+Wang%2C+and+Pengfei+Yin%2E+2020%2E+HIN%3A+hierarchical+inference+network+for+document%2Dlevel+relation+extraction%2E+In+Proceedings+of+KDD%2E)
18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of NIPS.
19. Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. [Simultaneously self-attending to all mentions for full-abstract biological relation extraction](https://md.research.elamur.ai/free/en/papers/verga-2018-simultaneously-self-attending-all-mentions/). In Proceedings of NAACL.
20. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of EMNLP: System Demonstrations.
21. Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Benfeng+Xu%2C+Quan+Wang%2C+Yajuan+Lyu%2C+Yong+Zhu%2C+and+Zhendong+Mao%2E+2021%2E+Entity+structure+within+and+throughout%3A+Modeling+mention+dependencies+for+document%2Dlevel+relation+extraction%2E+In+Proceedings+of+AAA)
22. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: a large-scale document-level relation extraction dataset. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Yuan+Yao%2C+Deming+Ye%2C+Peng+Li%2C+Xu+Han%2C+Yankai+Lin%2C+Zhenghao+Liu%2C+Zhiyuan+Liu%2C+Lixin+Huang%2C+Jie+Zhou%2C+and+Maosong+Sun%2E+2019%2E+DocRED%3A+a+large%2Dscale+document%2Dlevel+relation+extraction+dataset%2E+In+Proceedi)
23. Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. [Coreferential reasoning learning for language representation](https://md.research.elamur.ai/free/en/papers/ye-2020-coreferential-reasoning-learning-language-representation/). In Proceedings of EMNLP.
24. Shuang Zeng, Yuting Wu, and Baobao Chang. 2021. SIRE: Separate intra- and inter-sentential reasoning for document-level relation extraction. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Shuang+Zeng%2C+Yuting+Wu%2C+and+Baobao+Chang%2E+2021%2E+SIRE%3A+Separate+intra%2D+and+inter%2Dsentential+reasoning+for+document%2Dlevel+relation+extraction%2E+In+Findings+of+ACL%2E)
25. Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. [Double graph based reasoning for document-level relation extraction](https://md.research.elamur.ai/free/en/papers/zeng-2020-double-graph-based-reasoning-document/). In Proceedings of EMNLP.
26. Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In Proceedings of IJCAI. [find](https://scholar.google.com/scholar?q=Ningyu+Zhang%2C+Xiang+Chen%2C+Xin+Xie%2C+Shumin+Deng%2C+Chuanqi+Tan%2C+Mosha+Chen%2C+Fei+Huang%2C+Luo+Si%2C+and+Huajun+Chen%2E+2021%2E+Document%2Dlevel+relation+extraction+as+semantic+segmentation%2E+In+Proceedings+of+IJCAI%2E)
27. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. [Graph convolution over pruned dependency trees improves relation extraction](https://md.research.elamur.ai/free/en/papers/zhang-2018-graph-convolution-over-pruned-dependency/). In Proceedings of EMNLP.
28. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+EMNLP%2E)
29. Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Wenxuan+Zhou%2C+Kevin+Huang%2C+Tengyu+Ma%2C+and+Jing+Huang%2E+2021%2E+Document%2Dlevel+relation+extraction+with+adaptive+thresholding+and+localized+context+pooling%2E+In+Proceedings+of+AAAI%2E)
30. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kr%C3%B6tzsch%2E+2014%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2E)
31. Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019. Fine-tune BERT for DocRED with two-step process. arXiv preprint arXiv:1909.11898. — [arXiv:1909.11898](https://arxiv.org/abs/1909.11898) [find](https://scholar.google.com/scholar?q=Hong+Wang%2C+Christfried+Focke%2C+Rob+Sylvester%2C+Nilesh+Mishra%2C+and+William+Wang%2E+2019%2E+Fine%2Dtune+BERT+for+DocRED+with+two%2Dstep+process%2E+arXiv+preprint+arXiv%3A1909%2E11898%2E+%E2%80%94+%5BarXiv%3A1909%2E11898%5D%28https%3A%2F%2Farxiv)
32. Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. 2020. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In Proceedings of ECCV. [find](https://scholar.google.com/scholar?q=Huiyu+Wang%2C+Yukun+Zhu%2C+Bradley+Green%2C+Hartwig+Adam%2C+Alan+Yuille%2C+and+Liang%2DChieh+Chen%2E+2020%2E+Axial%2Ddeeplab%3A+Stand%2Dalone+axial%2Dattention+for+panoptic+segmentation%2E+In+Proceedings+of+ECCV%2E)

*Updated 24 October 2026*