---
title: "基于语言模型增强的中文关系抽取方法 — 薛平 2023"
description: "Chinese relation extraction (RE) classifies the semantic relation between two entities in a sentence."
authors: ["薛平", "李影", "吴中海"]
year: 2023
venue: "中文信息学报 (Journal of Chinese Information Processing)"
source: "http://jcip.cipsc.org.cn/article/id/zwxxxb_3558"
licence: "Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/xue-2023-language-model-enhanced/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/xue-2023-language-model-enhanced/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/7/) / 基于语言模型增强的中文关系抽取方法

# 基于语言模型增强的中文关系抽取方法

**Authors:** 薛平, 李影, 吴中海 · **Year:** 2023 · **Venue:** 中文信息学报 (Journal of Chinese Information Processing) · **Licence:** Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报

**Links:** [Publisher's page](http://jcip.cipsc.org.cn/article/id/zwxxxb_3558) · [PDF](http://jcip.cipsc.org.cn/cn/article/pdf/preview/zwxxxb_3558.pdf)

## Summary
Chinese relation extraction (RE) classifies the semantic relation between two entities in a sentence. The best results come from fine-tuned pre-trained language models, but those models have very large parameter counts and are expensive to store and run. The authors propose a cheaper alternative based on multi-task learning. A lightweight character-level BiLSTM relation classifier also trains a bidirectional language model that shares its embedding and encoder layers. Ordinary language models learn to predict the next character. Here, the language model instead learns to match the masked-character probability distributions produced by BERT-wwm, using KL divergence as the loss. This transfers general linguistic knowledge from BERT-wwm into the small model. The authors frame the effect as a form of automatic data augmentation, since the model is exposed to alternative phrasings that express the same relation. The method was evaluated on SanWen, ACE2004 and ACE2005. It beat lexicon-based approaches and reached about 95% of the pre-trained model's performance with roughly 1% of its parameters. This makes it attractive where compute is limited.

## Key points
- **Architecture:** The base model combines 100-dimensional pre-trained character embeddings with 10-dimensional relative-position embeddings for the head and tail entities. These feed a BiLSTM with hidden size 200 per direction, followed by a softmax classifier. The language-model heads reuse the shared encoder.
- **Teacher signal:** BERT-wwm (whole-word masking, 12-layer Transformer, pre-trained on Chinese Wikipedia) supplies soft labels for each masked character. These are truncated to the top 100 candidates and renormalized.
- **Training objective:** The total loss is the RE cross-entropy plus λ times the forward and backward KL terms, with λ = 0.01. Training used Adam with a learning rate of 0.0005 and dropout of 0.5.
- **Data:**
  - SanWen: literary texts, 10 relation types.
  - ACE2005: news, 18 relation types.
  - ACE2004: news, 19 relation types.
  - The ACE corpora use a 75/25 split, following Li et al.
  - Results are reported as F1 and AUC.
- **Baselines and positioning:** The method was compared with a CNN model (Zeng et al.), an LSTM model (Zhang et al.), MG-Lattice and Flat-Lattice. It differs from knowledge distillation because it learns task-independent language knowledge from a pre-trained model, rather than from a task-specific RE teacher.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{2023,
  title = {{基于语言模型增强的中文关系抽取方法}},
  author = {薛平 and 李影 and 吴中海},
  journal = {中文信息学报 (Journal of Chinese Information Processing)},
  year = {2023},
  url = {http://jcip.cipsc.org.cn/article/id/zwxxxb_3558}
}
```

- [← previous 基于神经网络模型的汉语文本难度分级](https://md.research.elamur.ai/free/papers/li-2023-chinese-readability-assessment/)
- [next → 对话式论辩研究综述](https://md.research.elamur.ai/free/papers/wei-2023-dialogical-argumentation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/7/)