---
title: "K-BERT: Enabling Language Representation with — Liu 2019"
description: "BERT-style models learn general language from open-domain text but lack the specialist knowledge needed for fields like medicine, finance, or law."
authors: ["Weijie Liu", "Peng Zhou", "Zhe Zhao", "Zhiruo Wang", "Qi Ju", "Haotang Deng", "Ping Wang"]
year: 2019
venue: "arXiv:1909.07606"
source: "https://arxiv.org/abs/1909.07606"
doi: "10.48550/arxiv.1909.07606"
arxiv: "1909.07606"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / K-BERT: Enabling Language Representation with Knowledge Graph

# K-BERT: Enabling Language Representation with Knowledge Graph

**Authors:** [Weijie Liu](https://md.research.elamur.ai/free/en/authors/weijie-liu/), [Peng Zhou](https://md.research.elamur.ai/free/en/authors/peng-zhou/), [Zhe Zhao](https://md.research.elamur.ai/free/en/authors/zhe-zhao/), [Zhiruo Wang](https://md.research.elamur.ai/free/en/authors/zhiruo-wang/), [Qi Ju](https://md.research.elamur.ai/free/en/authors/qi-ju/), [Haotang Deng](https://md.research.elamur.ai/free/en/authors/haotang-deng/), Ping Wang · **Year:** 2019 · **Venue:** arXiv:1909.07606 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1909.07606) · [PDF](https://arxiv.org/pdf/1909.07606) · [DOI 10.48550/arxiv.1909.07606](https://doi.org/10.48550/arxiv.1909.07606) · [arXiv 1909.07606](https://arxiv.org/abs/1909.07606) · [OpenAlex](https://openalex.org/W2998385486)

**Citations:** 808 citations (openalex-snapshot · 2026-10-14) · [198](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
BERT-style models learn general language from open-domain text but lack the specialist knowledge needed for fields like medicine, finance, or law. Pre-training a domain-specific model is costly. K-BERT addresses this by inserting relevant knowledge graph triples directly into input sentences at fine-tuning time. A knowledge layer looks up entities in a KG and attaches their triples, producing a sentence tree with depth 1. Too much injected knowledge can distort a sentence's meaning, a problem the authors call knowledge noise. To prevent this, K-BERT uses two mechanisms. Soft-position embeddings preserve the original word order. A visible matrix inside a mask-transformer limits which tokens can attend to the added triples. Because K-BERT keeps BERT's parameters unchanged, any pre-trained BERT checkpoint can be loaded without extra pre-training. On twelve Chinese NLP tasks, using HowNet, CN-DBpedia and a MedicalKG, gains were small on open-domain tasks but consistent on domain-specific ones. This makes K-BERT a cheap way to add expert knowledge to language models.

## Key points
- **Architecture:** K-BERT has four components: a knowledge layer, an embedding layer (token, soft-position and segment embeddings), a seeing layer that builds the visible matrix, and a mask-transformer encoder.
- **Domain benchmarks:** Finance Q&A and Law Q&A were built from data crawled from Baidu Zhidao, roughly 770k and 36k samples. Finance NER covers 3,000 news articles. Medicine NER is the CCKS 2017 CNER task.
- **F1 gains over Google BERT:**
  - Medicine NER: 92.5 → 94.2 (MedicalKG)
  - Law Q&A: 86.4 → 87.5 (CN-DBpedia)
  - Finance NER: 86.1 → 87.6 (HowNet)
- **Task-dependent effects:** KGs gave no clear benefit on sentiment tasks. HowNet outperformed the encyclopedic KG on semantic similarity.
- **Ablation:** Without the visible matrix, K-BERT scored below plain BERT on Law Q&A, which confirms the knowledge noise problem. K-BERT also peaked earlier (epoch 2 vs. 4). Code and KGs are released at github.com/autoliuweijie/K-BERT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 191 not yet.

- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Integrating Transformers and Knowledge Graphs for Twitter Stance Detection](https://md.research.elamur.ai/free/en/papers/clark-2021-integrating-transformers-knowledge-graphs-twitter/)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2019k,
  title = {{K-BERT: Enabling Language Representation with Knowledge Graph}},
  author = {Liu, Weijie and Zhou, Peng and Zhao, Zhe and Wang, Zhiruo and Ju, Qi and Deng, Haotang and Wang, Ping},
  year = {2019},
  doi = {10.48550/arxiv.1909.07606},
  eprint = {1909.07606},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.07606}
}
```

- [← previous JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/agic-2019-jw300-wide-coverage-parallel-corpus/)
- [next → Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories](https://md.research.elamur.ai/free/en/papers/zarrie-2019-know-what-you-don-t/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bodenreider, O. 2008. Biomedical ontologies in action: role in knowledge management, data integration and decision support. Yearbook of medical informatics 17(01):67– 79. [find](https://scholar.google.com/scholar?q=Bodenreider%2C+O%2E+2008%2E+Biomedical+ontologies+in+action%3A+role+in+knowledge+management%2C+data+integration+and+decision+support%2E+Yearbook+of+medical+informatics+17%2801%29%3A67%E2%80%93+79%2E)
2. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, 2787–2795. [find](https://scholar.google.com/scholar?q=Bordes%2C+A%2E%3B+Usunier%2C+N%2E%3B+Garcia%2DDuran%2C+A%2E%3B+Weston%2C+J%2E%3B+and+Yakhnenko%2C+O%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+Advances+in+neural+information+processing+systems%2C+2787%E2%80%93279)
3. Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317. [find](https://scholar.google.com/scholar?q=Bosselut%2C+A%2E%3B+Rashkin%2C+H%2E%3B+Sap%2C+M%2E%3B+Malaviya%2C+C%2E%3B+Celikyilmaz%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+COMET%3A+Commonsense+transformers+for+automatic+knowledge+graph+construction%2E+arXiv+preprint+arXiv%3A1906%2E05317%2E) [our card](https://md.research.elamur.ai/free/en/papers/bosselut-2019-comet-commonsense-transformers-automatic-knowledge/)
4. Cao, Y.; Hou, L.; Li, J.; Liu, Z.; Li, C.; Chen, X.; and Dong, T. 2018. Joint representation learning of cross-lingual words and entities via attentive distant supervision. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Cao%2C+Y%2E%3B+Hou%2C+L%2E%3B+Li%2C+J%2E%3B+Liu%2C+Z%2E%3B+Li%2C+C%2E%3B+Chen%2C+X%2E%3B+and+Dong%2C+T%2E+2018%2E+Joint+representation+learning+of+cross%2Dlingual+words+and+entities+via+attentive+distant+supervision%2E+In+Proceedings+of+the+2018+)
5. Conneau, A.; Lample, G.; Rinott, R.; Williams, A.; Bowman, S. R.; Schwenk, H.; and Stoyanov, V. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%3B+Lample%2C+G%2E%3B+Rinott%2C+R%2E%3B+Williams%2C+A%2E%3B+Bowman%2C+S%2E+R%2E%3B+Schwenk%2C+H%2E%3B+and+Stoyanov%2C+V%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the+2018+Conference+on+)
6. Cui, Y.; Che, W.; Liu, T.; Qin, B.; Yang, Z.; Wang, S.; and Hu, G. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101. [find](https://scholar.google.com/scholar?q=Cui%2C+Y%2E%3B+Che%2C+W%2E%3B+Liu%2C+T%2E%3B+Qin%2C+B%2E%3B+Yang%2C+Z%2E%3B+Wang%2C+S%2E%3B+and+Hu%2C+G%2E+2019%2E+Pre%2Dtraining+with+whole+word+masking+for+chinese+bert%2E+arXiv+preprint+arXiv%3A1906%2E08101%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/)
7. Dai, Z.; Yang, Z.; Yang, Y.; Cohen, W. W.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%3B+Yang%2C+Z%2E%3B+Yang%2C+Y%2E%3B+Cohen%2C+W%2E+W%2E%3B+Carbonell%2C+J%2E%3B+Le%2C+Q%2E+V%2E%3B+and+Salakhutdinov%2C+R%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E0286)
8. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%3B+Chang%2C+M%2E%2DW%2E%3B+Lee%2C+K%2E%3B+and+Toutanova%2C+K%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
9. Dong, Z.; Dong, Q.; and Hao, C. 2006. Hownet and the computation of meaning. Citeseer. [find](https://scholar.google.com/scholar?q=Dong%2C+Z%2E%3B+Dong%2C+Q%2E%3B+and+Hao%2C+C%2E+2006%2E+Hownet+and+the+computation+of+meaning%2E+Citeseer%2E)
10. Han, X.; Liu, Z.; and Sun, M. 2016. Joint representation learning of text and knowledge for knowledge graph completion. arXiv preprint arXiv:1611.04125. — [arXiv:1611.04125](https://arxiv.org/abs/1611.04125) [find](https://scholar.google.com/scholar?q=Han%2C+X%2E%3B+Liu%2C+Z%2E%3B+and+Sun%2C+M%2E+2016%2E+Joint+representation+learning+of+text+and+knowledge+for+knowledge+graph+completion%2E+arXiv+preprint+arXiv%3A1611%2E04125%2E+%E2%80%94+%5BarXiv%3A1611%2E04125%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611)
11. Joshi, M.; Chen, D.; Liu, Y.; Weld, D. S.; Zettlemoyer, L.; and Levy, O. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%3B+Chen%2C+D%2E%3B+Liu%2C+Y%2E%3B+Weld%2C+D%2E+S%2E%3B+Zettlemoyer%2C+L%2E%3B+and+Levy%2C+O%2E+2019%2E+SpanBERT%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+arXiv+preprint+arXiv%3A1907%2E10529%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
12. Levow, G. A. 2006. The 3rd international chinese language processing bakeoff. In Sighan Workshop on Chinese Language Processing. [find](https://scholar.google.com/scholar?q=Levow%2C+G%2E+A%2E+2006%2E+The+3rd+international+chinese+language+processing+bakeoff%2E+In+Sighan+Workshop+on+Chinese+Language+Processing%2E)
13. Liu, X.; Chen, Q.; Deng, C.; Zeng, H.; Chen, J.; Li, D.; and Tang, B. 2018. LCQMC:a large-scale chinese question matching corpus. International conference on computational linguistics 1952–1962. [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%3B+Chen%2C+Q%2E%3B+Deng%2C+C%2E%3B+Zeng%2C+H%2E%3B+Chen%2C+J%2E%3B+Li%2C+D%2E%3B+and+Tang%2C+B%2E+2018%2E+LCQMC%3Aa+large%2Dscale+chinese+question+matching+corpus%2E+International+conference+on+computational+linguistics+1952%E2%80%931962%2E)
14. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%3B+Ott%2C+M%2E%3B+Goyal%2C+N%2E%3B+Du%2C+J%2E%3B+Joshi%2C+M%2E%3B+Chen%2C+D%2E%3B+Levy%2C+O%2E%3B+Lewis%2C+M%2E%3B+Zettlemoyer%2C+L%2E%3B+and+Stoyanov%2C+V%2E+2019%2E)
15. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [find](https://scholar.google.com/scholar?q=RoBERTa%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+preprint+arXiv%3A1907%2E11692%2E+Mikolov%2C+T%2E%3B+Chen%2C+K%2E%3B+Corrado%2C+G%2E%3B+and+Dean%2C+J%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+)
16. Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%3B+Neumann%2C+M%2E%3B+Iyyer%2C+M%2E%3B+Gardner%2C+M%2E%3B+Clark%2C+C%2E%3B+Lee%2C+K%2E%3B+and+Zettlemoyer%2C+L%2E+2018%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2E)
17. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%3B+Narasimhan%2C+K%2E%3B+Salimans%2C+T%2E%3B+and+Sutskever%2C+I%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+report%2C+OpenAI%2E)
18. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; and Wu, H. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. [find](https://scholar.google.com/scholar?q=Sun%2C+Y%2E%3B+Wang%2C+S%2E%3B+Li%2C+Y%2E%3B+Feng%2C+S%2E%3B+Chen%2C+X%2E%3B+Zhang%2C+H%2E%3B+Tian%2C+X%2E%3B+Zhu%2C+D%2E%3B+Tian%2C+H%2E%3B+and+Wu%2C+H%2E+2019%2E+ERNIE%3A+Enhanced+representation+through+knowledge+integration%2E+arXiv+preprint+arXiv%3A1904%2E09223%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
19. Toutanova, K.; Chen, D.; Pantel, P.; Poon, H.; Choudhury, P.; and Gamon, M. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1499–1509. [find](https://scholar.google.com/scholar?q=Toutanova%2C+K%2E%3B+Chen%2C+D%2E%3B+Pantel%2C+P%2E%3B+Poon%2C+H%2E%3B+Choudhury%2C+P%2E%3B+and+Gamon%2C+M%2E+2015%2E+Representing+text+for+joint+embedding+of+text+and+knowledge+bases%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/toutanova-2015-representing-text-joint-embedding-text/)
20. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%3B+Shazeer%2C+N%2E%3B+Parmar%2C+N%2E%3B+Uszkoreit%2C+J%2E%3B+Jones%2C+L%2E%3B+Gomez%2C+A%2E+N%2E%3B+Kaiser%2C+%C5%81%2E%3B+and+Polosukhin%2C+I%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+systems%2C+5998)
21. Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014. Knowledge graph and text jointly embedding. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1591–1601. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%3B+Zhang%2C+J%2E%3B+Feng%2C+J%2E%3B+and+Chen%2C+Z%2E+2014%2E+Knowledge+graph+and+text+jointly+embedding%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+%28EMNLP%29%2C+1591%E2%80%9316)
22. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%3B+Singh%2C+A%2E%3B+Michael%2C+J%2E%3B+Hill%2C+F%2E%3B+Levy%2C+O%2E%3B+and+Bowman%2C+S%2E+R%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1804%2E07461%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
23. Xu, B.; Xu, Y.; Liang, J.; Xie, C.; Liang, B.; Cui, W.; and Xiao, Y. 2017. Cn-dbpedia: A never-ending chinese knowledge extraction system. International conference industrial, engineering and other applications applied intelligent systems 428–438. [find](https://scholar.google.com/scholar?q=Xu%2C+B%2E%3B+Xu%2C+Y%2E%3B+Liang%2C+J%2E%3B+Xie%2C+C%2E%3B+Liang%2C+B%2E%3B+Cui%2C+W%2E%3B+and+Xiao%2C+Y%2E+2017%2E+Cn%2Ddbpedia%3A+A+never%2Dending+chinese+knowledge+extraction+system%2E+International+conference+industrial%2C+engineering+and+other+ap)
24. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%3B+Dai%2C+Z%2E%3B+Yang%2C+Y%2E%3B+Carbonell%2C+J%2E%3B+Salakhutdinov%2C+R%2E%3B+and+Le%2C+Q%2E+V%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08237%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
25. Zhang, Z.; Han, X.; Liu, Z.; Jiang, X.; Sun, M.; and Liu, Q. 2019. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%3B+Han%2C+X%2E%3B+Liu%2C+Z%2E%3B+Jiang%2C+X%2E%3B+Sun%2C+M%2E%3B+and+Liu%2C+Q%2E+2019%2E+ERNIE%3A+Enhanced+language+representation+with+informative+entities%2E+arXiv+preprint+arXiv%3A1905%2E07129%2E)