---
title: "CoLAKE: Contextualized Language and Knowledge Embedding"
description: "Pre-trained language models such as BERT capture factual knowledge poorly. 42 references, cited by 58 papers here."
authors: ["Tianxiang Sun", "Yunfan Shao", "Xipeng Qiu", "Qipeng Guo", "Yaru Hu", "Xuanjing Huang", "Zheng Zhang"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.327/"
doi: "10.18653/v1/2020.coling-main.327"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sun-2020-colake-contextualized-language-knowledge-embedding/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2020-colake-contextualized-language-knowledge-embedding/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / CoLAKE: Contextualized Language and Knowledge Embedding

# CoLAKE: Contextualized Language and Knowledge Embedding

**Authors:** [Tianxiang Sun](https://md.research.elamur.ai/free/en/authors/tianxiang-sun/), [Yunfan Shao](https://md.research.elamur.ai/free/en/authors/yunfan-shao/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/), [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/), Yaru Hu, [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) and [Zheng Zhang](https://md.research.elamur.ai/free/en/authors/zheng-zhang/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.327/) · [PDF](https://aclanthology.org/2020.coling-main.327.pdf) · [DOI 10.18653/v1/2020.coling-main.327](https://doi.org/10.18653/v1/2020.coling-main.327) · [OpenAlex](https://openalex.org/W3114916066)

**Citations:** 180 citations (OpenAlex, 2026-10-14) · [58](https://md.research.elamur.ai/free/en/papers/sun-2020-colake-contextualized-language-knowledge-embedding/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Pre-trained language models such as BERT capture factual knowledge poorly. Earlier knowledge-enhanced models like ERNIE and KnowBERT tried to fix this by adding entity vectors, but those vectors are trained separately by knowledge embedding methods (e.g., TransE), stay frozen, and have to be retrained whenever the knowledge graph changes. CoLAKE takes a different route and learns language and knowledge representations together in one shared space.

For each entity mentioned in a sentence, it pulls a small neighbourhood of triplets from a knowledge base. That neighbourhood is merged with the fully connected token graph into a single heterogeneous structure called a word-knowledge (WK) graph. A lightly modified Transformer encoder, steered by the graph's adjacency matrix, processes this input. The masked language modelling objective is applied to word, entity and relation nodes alike.

On GLUE, CoLAKE performs almost on par with RoBERTa. On a new WK graph completion task, it far outperforms classic knowledge embedding models and can even handle entities never seen in training. This suggests that representing an entity through its context, rather than a fixed vector, is a promising way to combine text with structured knowledge.

## Key points

- **WK graph construction:**
  - Each mention is replaced by its linked entity, which becomes an "anchor" node.
  - Up to 15 randomly sampled neighbouring relations and entities per anchor form the injected knowledge sub-graph.
  - Relations are themselves represented as nodes.
- **GLUE (dev set, all models initialized from RoBERTa BASE):**
  - CoLAKE averages 86.3, against 86.4 for RoBERTa and 84.9 for KEPLER.
  - This indicates little loss in general language understanding.
- **WK graph completion, transductive setting (10K FewRel test samples, relation prediction):**
  - CoLAKE reaches MRR 82.48 and HITS@10 98.58, with mean rank 2.03.
  - The best baseline, RotatE, scores MRR 70.90.
  - The other baselines are TransE, DistMult and ComplEx, all trained via DGL-KE on Wikidata5M.
- **Inductive setting (1K samples with unseen entities):**
  - CoLAKE reaches MRR 28.10 and HITS@10 58.05, versus 8.18 and 14.13 for DKRL.
  - CoLAKE infers unseen entities from their graph neighbours rather than from textual descriptions.
- **Code and future directions:**
  - Code is released at github.com/txsun1997/CoLAKE.
  - Suggested future uses include cleaning distantly supervised relation extraction data and judging the quality of graph-to-text templates.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 44 works · 10,162 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang and Zheng Zhang. 2020. CoLAKE: Contextualized Language and Knowledge Embedding. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sun2020colake,
  title = {{CoLAKE: Contextualized Language and Knowledge Embedding}},
  author = {Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang and Zheng Zhang},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.327},
  url = {https://doi.org/10.18653/v1/2020.coling-main.327}
}
```

- [← previous Curriculum Learning for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/xu-2020-curriculum-learning-natural-language-understanding/)
- [next → COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://md.research.elamur.ai/free/en/papers/hwang-2020-comet-atomic-2020-symbolic-neural/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Alberto+Garc%C4%B1%CC%81a%2DDura%CC%81n%2C+Jason+Weston%2C+and+Oksana+Yakhnenko%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+NIPS%2E)
2. Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In AAAI, pages 7456–7463. [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose%CC%81+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2020%2E+Inducing+relational+knowledge+from+BERT%2E+In+AAAI%2C+pages+7456%E2%80%937463%2E)
3. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In ACL, pages 87–96. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+Omer+Levy%2C+Yejin+Choi%2C+and+Luke+Zettlemoyer%2E+2018%2E+Ultra%2Dfine+entity+typing%2E+In+ACL%2C+pages+87%E2%80%9396%2E)
4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, pages 160–167. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+deep+neural+networks+with+multitask+learning%2E+In+ICML%2C+pages+160%E2%80%93167%2E)
5. Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In ISMB, pages 77–86. [find](https://scholar.google.com/scholar?q=Mark+Craven+and+Johan+Kumlien%2E+1999%2E+Constructing+biological+knowledge+bases+by+extracting+information+from+text+sources%2E+In+ISMB%2C+pages+77%E2%80%9386%2E)
6. Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. [Commonsense knowledge mining from pretrained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In EMNLP-IJCNLP, pages 1173–1178.
7. Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In AAAI, pages 1811–1818. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Pasquale+Minervini%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2018%2E+Convolutional+2d+knowledge+graph+embeddings%2E+In+AAAI%2C+pages+1811%E2%80%931818%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL-HLT, pages 4171–4186.
9. Paolo Ferragina and Ugo Scaiella. 2010. [TAGME: on-the-fly annotation of short text fragments (by wikipedia entities)](https://md.research.elamur.ai/free/en/papers/ferragina-2010-tagme-fly-annotation-short/). In CIKM, pages 1625–1628.
10. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In EMNLP, pages 2619–2629. [find](https://scholar.google.com/scholar?q=Octavian%2DEugen+Ganea+and+Thomas+Hofmann%2E+2017%2E+Deep+joint+entity+disambiguation+with+local+neural+attention%2E+In+EMNLP%2C+pages+2619%E2%80%932629%2E)
11. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In EMNLP, pages 4803–4809. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Hao+Zhu%2C+Pengfei+Yu%2C+Ziyun+Wang%2C+Yuan+Yao%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2018%2E+Fewrel%3A+A+large%2Dscale+supervised+few%2Dshot+relation+classification+dataset+with+state%2Dof%2Dthe%2Dart+evaluation%2E+In+EM)
12. Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. 2019. Integrating graph contextualized knowledge into pre-trained language models. CoRR, abs/1912.00147. — [arXiv:1912.00147](https://arxiv.org/abs/1912.00147) [find](https://scholar.google.com/scholar?q=Bin+He%2C+Di+Zhou%2C+Jinghui+Xiao%2C+Xin+Jiang%2C+Qun+Liu%2C+Nicholas+Jing+Yuan%2C+and+Tong+Xu%2E+2019%2E+Integrating+graph+contextualized+knowledge+into+pre%2Dtrained+language+models%2E+CoRR%2C+abs%2F1912%2E00147%2E+%E2%80%94+%5BarXiv%3A19)
13. Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI, pages 2181–2187. [find](https://scholar.google.com/scholar?q=Yankai+Lin%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+Yang+Liu%2C+and+Xuan+Zhu%2E+2015%2E+Learning+entity+and+relation+embeddings+for+knowledge+graph+completion%2E+In+AAAI%2C+pages+2181%E2%80%932187%2E)
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
15. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. [K-BERT: enabling language representation with knowledge graph](https://md.research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/).
16. Robert L. Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In ACL, pages 5962–5971. [find](https://scholar.google.com/scholar?q=Robert+L%2E+Logan%2C+Nelson+F%2E+Liu%2C+Matthew+E%2E+Peters%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Barack%E2%80%99s+wife+hillary%3A+Using+knowledge+graphs+for+fact%2Daware+language+modeling%2E+In+ACL%2C+pages+5962%E2%80%935971%2E)
17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In NIPS, pages 3111–3119.
18. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In ACL, pages 1003–1011. [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Daniel+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+ACL%2C+pages+1003%E2%80%931011%2E)
19. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP, pages 1532–1543.
20. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL-HLT, pages 2227–2237.
21. Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP-IJCNLP, pages 43–54. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Robert+L%2E+Logan+IV%2C+Roy+Schwartz%2C+Vidur+Joshi%2C+Sameer+Singh%2C+and+Noah+A%2E+Smith%2E+2019%2E+Knowledge+enhanced+contextual+word+representations%2E+In+EMNLP%2DIJCNLP%2C+pages+43%E2%80%9354%2E)
22. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In EMNLP-IJCNLP, pages 2463–2473.
23. Nina Pörner, Ulli Waltinger, and Hinrich Schütze. 2019. BERT is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised QA. CoRR, abs/1911.03681. [find](https://scholar.google.com/scholar?q=Nina+Po%CC%88rner%2C+Ulli+Waltinger%2C+and+Hinrich+Schu%CC%88tze%2E+2019%2E+BERT+is+not+a+knowledge+base+%28yet%29%3A+Factual+knowledge+vs%2E+name%2Dbased+reasoning+in+unsupervised+QA%2E+CoRR%2C+abs%2F1911%2E03681%2E)
24. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. [Pre-trained models for natural language processing: A survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/). CoRR, abs/2003.08271.
25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In ACL.
26. Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Zhi%2DHong+Deng%2C+Jian%2DYun+Nie%2C+and+Jian+Tang%2E+2019%2E+Rotate%3A+Knowledge+graph+embedding+by+relational+rotation+in+complex+space%2E+In+ICLR%2E)
27. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML. [find](https://scholar.google.com/scholar?q=The%CC%81o+Trouillon%2C+Johannes+Welbl%2C+Sebastian+Riedel%2C+E%CC%81ric+Gaussier%2C+and+Guillaume+Bouchard%2E+2016%2E+Complex+embeddings+for+simple+link+prediction%2E+In+ICML%2E)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, pages 5998–6008.
29. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In ICLR. [find](https://scholar.google.com/scholar?q=Petar+Velickovic%2C+Guillem+Cucurull%2C+Arantxa+Casanova%2C+Adriana+Romero%2C+Pietro+Lio%CC%80%2C+and+Yoshua+Bengio%2E+2018%2E+Graph+attention+networks%2E+In+ICLR%2E)
30. Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. [Knowledge graph and text jointly embedding](https://md.research.elamur.ai/free/en/papers/wang-2014-knowledge-graph-text-jointly-embedding/). In EMNLP, pages 1591–1601.
31. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In ICLR.
32. Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. 2019b. Coke: Contextualized knowledge graph embedding. CoRR, abs/1911.02168. — [arXiv:1911.02168](https://arxiv.org/abs/1911.02168) [find](https://scholar.google.com/scholar?q=Quan+Wang%2C+Pingping+Huang%2C+Haifeng+Wang%2C+Songtai+Dai%2C+Wenbin+Jiang%2C+Jing+Liu%2C+Yajuan+Lyu%2C+Yong+Zhu%2C+and+Hua+Wu%2E+2019b%2E+Coke%3A+Contextualized+knowledge+graph+embedding%2E+CoRR%2C+abs%2F1911%2E02168%2E+%E2%80%94+%5BarXiv%3A19)
33. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019c. KEPLER: A unified model for knowledge embedding and pre-trained language representation. CoRR, abs/1911.06136. [find](https://scholar.google.com/scholar?q=Xiaozhi+Wang%2C+Tianyu+Gao%2C+Zhaocheng+Zhu%2C+Zhiyuan+Liu%2C+Juanzi+Li%2C+and+Jian+Tang%2E+2019c%2E+KEPLER%3A+A+unified+model+for+knowledge+embedding+and+pre%2Dtrained+language+representation%2E+CoRR%2C+abs%2F1911%2E06136%2E)
34. Hongwei Wang, Hongyu Ren, and Jure Leskovec. 2020a. Entity context and relational paths for knowledge graph completion. CoRR, abs/2002.06757. — [arXiv:2002.06757](https://arxiv.org/abs/2002.06757) [find](https://scholar.google.com/scholar?q=Hongwei+Wang%2C+Hongyu+Ren%2C+and+Jure+Leskovec%2E+2020a%2E+Entity+context+and+relational+paths+for+knowledge+graph+completion%2E+CoRR%2C+abs%2F2002%2E06757%2E+%E2%80%94+%5BarXiv%3A2002%2E06757%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E06757%29)
35. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020b. [K-adapter: Infusing knowledge into pre-trained models with adapters](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/). CoRR, abs/2002.01808.
36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). ArXiv, abs/1910.03771.
37. Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation learning of knowledge graphs with entity descriptions. In AAAI. [find](https://scholar.google.com/scholar?q=Ruobing+Xie%2C+Zhiyuan+Liu%2C+Jia+Jia%2C+Huanbo+Luan%2C+and+Maosong+Sun%2E+2016%2E+Representation+learning+of+knowledge+graphs+with+entity+descriptions%2E+In+AAAI%2E)
38. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. [Joint learning of the embedding of words and entities for named entity disambiguation](https://md.research.elamur.ai/free/en/papers/yamada-2016-joint-learning-embedding-words-entities/). In CoNLL, pages 250–259.
39. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. [Embedding entities and relations for learning and inference in knowledge bases](https://md.research.elamur.ai/free/en/papers/yang-2014-embedding-entities-relations-learning-inference/). In ICLR.
40. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NeurIPS, pages 5754–5764.
41. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In ACL, pages 1441–1451. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+ERNIE%3A+enhanced+language+representation+with+informative+entities%2E+In+ACL%2C+pages+1441%E2%80%931451%2E)
42. Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis. 2020. DGL-KE: training knowledge graph embeddings at scale. CoRR, abs/2004.08532. [find](https://scholar.google.com/scholar?q=Da+Zheng%2C+Xiang+Song%2C+Chao+Ma%2C+Zeyuan+Tan%2C+Zihao+Ye%2C+Jin+Dong%2C+Hao+Xiong%2C+Zheng+Zhang%2C+and+George+Karypis%2E+2020%2E+DGL%2DKE%3A+training+knowledge+graph+embeddings+at+scale%2E+CoRR%2C+abs%2F2004%2E08532%2E)

*Updated 24 October 2026*