---
title: "ZS-BERT — paper by Chen and Li (2021) — summary and ideas"
description: "A BERT-based encoder builds a sentence embedding by combining the [CLS] vector with… 13.54% F1. 20 references, cited by 51 papers here."
authors: ["Chih-Yao Chen", "Cheng-Te Li"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.272/"
doi: "10.18653/v1/2021.naacl-main.272"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2021-zs-bert-zero-shot-relation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2021-zs-bert-zero-shot-relation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

# ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

**Authors:** Chih-Yao Chen and [Cheng-Te Li](https://md.research.elamur.ai/free/en/authors/cheng-te-li/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.272/) · [PDF](https://aclanthology.org/2021.naacl-main.272.pdf) · [DOI 10.18653/v1/2021.naacl-main.272](https://doi.org/10.18653/v1/2021.naacl-main.272) · [OpenAlex](https://openalex.org/W3172497214)

**Citations:** 91 citations (OpenAlex, 2026-10-14) · [51](https://md.research.elamur.ai/free/en/papers/chen-2021-zs-bert-zero-shot-relation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most relation extraction systems can only recognise relation types that appear in their training data, yet new relations keep emerging. Chen and Li address zero-shot relation extraction, where the relation labels seen at test time never occur during training. Their model, ZS-BERT, needs only a short textual description of each relation, which can usually be taken from open resources. It does not need manually written question templates or annotated class attributes. A BERT-based encoder builds a sentence embedding by combining the [CLS] vector with the representations of the two target entities. Sentence-BERT encodes the relation descriptions. Training is multi-task: each sentence embedding is pulled toward the embedding of its relation's description, and the model also classifies the seen relations. At test time, descriptions of unseen relations are embedded, and each new sentence gets the label of the nearest relation vector. On two well-known benchmarks, ZS-BERT outperforms earlier zero-shot methods by at least 13.54% F1. The work shows that cheap relation descriptions can replace costly annotation when handling newly appearing relations.

## Key points
- **Architecture:** The [CLS] vector and two entity representations are concatenated and passed through a fully-connected layer with activation to form the sentence attribute vector. Relation descriptions are embedded with Sentence-BERT. A ranking-style distance loss is combined with seen-relation classification.
- **Baselines:**
  - Levy et al. (2017) cast the task as question answering with 10 hand-defined templates.
  - Obamuyide and Vlachos (2018) treat it as textual entailment using ESIM and CIM. This yields only a yes/no decision per sentence–description pair.
  - R-BERT is the strongest embedding-based competitor using nearest-neighbour search.
- **Main result:** ZS-BERT gains at least 13.54% F1 over existing methods. It can also be adapted to few-shot settings when a small amount of labelled data for unseen relations is available.
- **Analysis:**
  - t-SNE plots show that ZS-BERT separates relation clusters more clearly than R-BERT.
  - Semantically close relations (e.g., subsidiary vs. owned by) still overlap.
  - The error analysis groups mistakes into four types, including errors that stem from distant-supervision labels.
- **Code:** Available at https://github.com/dinobby/ZS-BERT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Taiwan University](https://md.research.elamur.ai/free/en/universities/national-taiwan-university/) — University, Taipei, Taiwan · 13 works · 1,886 citations
- [National Cheng Kung University](https://md.research.elamur.ai/free/en/universities/national-cheng-kung-university/) — University, Tainan, Taiwan · 3 works · 613 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works

## Cite

Cite as: Chih-Yao Chen and Cheng-Te Li. 2021. ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2021zs,
  title = {{ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning}},
  author = {Chih-Yao Chen and Cheng-Te Li},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.272},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.272}
}
```

- [← previous Towards Robustness of Text-to-SQL Models against Synonym Substitution](https://md.research.elamur.ai/free/en/papers/gan-2021-robustness-text-sql-models-against/)
- [next → Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://md.research.elamur.ai/free/en/papers/gu-2021-fully-non-autoregressive-neural-machine/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. E. Kodirov, T. Xiang, and S. Gong. 2017. Semantic autoencoder for zero-shot learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4447–4456. [find](https://scholar.google.com/scholar?q=E%2E+Kodirov%2C+T%2E+Xiang%2C+and+S%2E+Gong%2E+2017%2E+Semantic+autoencoder+for+zero%2Dshot+learning%2E+In+2017+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+4447%E2%80%934456%2E)
2. J. L. Ba, K. Swersky, S. Fidler, and R. Salakhutdinov. 2015. Predicting deep zero-shot convolutional neural networks using textual descriptions. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 4247–4255. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Ba%2C+K%2E+Swersky%2C+S%2E+Fidler%2C+and+R%2E+Salakhutdinov%2E+2015%2E+Predicting+deep+zero%2Dshot+convolutional+neural+networks+using+textual+descriptions%2E+In+2015+IEEE+International+Conference+on+Computer+Visio)
3. C. H. Lampert, H. Nickisch, and S. Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951–958. [find](https://scholar.google.com/scholar?q=C%2E+H%2E+Lampert%2C+H%2E+Nickisch%2C+and+S%2E+Harmeling%2E+2009%2E+Learning+to+detect+unseen+object+classes+by+between%2Dclass+attribute+transfer%2E+In+2009+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pa)
4. C. H. Lampert, H. Nickisch, and S. Harmeling. 2014. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453–465. [find](https://scholar.google.com/scholar?q=C%2E+H%2E+Lampert%2C+H%2E+Nickisch%2C+and+S%2E+Harmeling%2E+2014%2E+Attribute%2Dbased+classification+for+zero%2Dshot+visual+object+categorization%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+36%283%29%3A453)
5. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Luke+Zettlemoyer%2E+2017%2E+Zero%2Dshot+relation+extraction+via+reading+comprehension%2E+In+Proceedings+of+the+21st+Conference+on+Computational+Natural+Language+Learni)
6. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence. [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+Yang+Liu%2C+and+Xuan+Zhu%2E+2015%2E+Learning+entity+and+relation+embeddings+for+knowledge+graph+completion%2E+In+Twenty%2Dninth+AAAI+conference+on+artificial+intelligence%2E)
7. Jiang Lu, Jin Li, Ziang Yan, Fenghua Mei, and Changshui Zhang. 2018. Attribute-based synthetic network (abs-net): Learning more from pseudo feature representations. Pattern Recognition, 80:129–142. [find](https://scholar.google.com/scholar?q=Jiang+Lu%2C+Jin+Li%2C+Ziang+Yan%2C+Fenghua+Mei%2C+and+Changshui+Zhang%2E+2018%2E+Attribute%2Dbased+synthetic+network+%28abs%2Dnet%29%3A+Learning+more+from+pseudo+feature+representations%2E+Pattern+Recognition%2C+80%3A129%E2%80%93142%2E)
8. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
9. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages… [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Daniel+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+t)
10. Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings. [find](https://scholar.google.com/scholar?q=Mohammad+Norouzi%2C+Tomas+Mikolov%2C+Samy+Bengio%2C+Yoram+Singer%2C+Jonathon+Shlens%2C+Andrea+Frome%2C+Greg+S%2E+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Zero%2Dshot+learning+by+convex+combination+of+semantic+embeddings%2E)
11. Abiola Obamuyide and Andreas Vlachos. 2018. Zero-shot relation classification as textual entailment. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 72–78, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abiola+Obamuyide+and+Andreas+Vlachos%2E+2018%2E+Zero%2Dshot+relation+classification+as+textual+entailment%2E+In+Proceedings+of+the+First+Workshop+on+Fact+Extraction+and+VERification+%28FEVER%29%2C+pages+72%E2%80%9378%2C+Brus)
12. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
14. Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1784–1789, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daniil+Sorokin+and+Iryna+Gurevych%2E+2017%2E+Context%2Daware+representations+for+knowledge+base+relation+extraction%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing)
15. Shanchan Wu and Yifan He. 2019. Enriching pre-trained language model with entity information for relation classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, page 2361–2364, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Shanchan+Wu+and+Yifan+He%2E+2019%2E+Enriching+pre%2Dtrained+language+model+with+entity+information+for+relation+classification%2E+In+Proceedings+of+the+28th+ACM+International+Conference+on+Information+and+Kno)
16. Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. [Question answering on Freebase via relation extraction and textual evidence](https://md.research.elamur.ai/free/en/papers/xu-2016-question-answering-freebase-relation-extraction/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2326–2336, Berlin, Germany.…
17. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. [Relation classification via convolutional deep neural network](https://md.research.elamur.ai/free/en/papers/zeng-2014-relation-classification-convolutional-deep-neural/). In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344, Dublin, Ireland. Dublin City Universi…
18. Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification. In Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation, pages 73–78, Shanghai, China. [find](https://scholar.google.com/scholar?q=Shu+Zhang%2C+Dequan+Zheng%2C+Xinchen+Hu%2C+and+Ming+Yang%2E+2015%2E+Bidirectional+long+short%2Dterm+memory+networks+for+relation+classification%2E+In+Proceedings+of+the+29th+Pacific+Asia+Conference+on+Language%2C+Inf)
19. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. [Attention-based bidirectional long short-term memory networks for relation classification](https://md.research.elamur.ai/free/en/papers/zhou-2016-attention-based-bidirectional-long-short/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20…
20. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015. [Reasoning about entailment with neural attention](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/). arXiv preprint arXiv:1509.06664.

*Updated 24 October 2026*