---
title: "Simple BERT Models for Relation Extraction and — Shi 2019"
description: "Shi and Lin examine whether pretrained BERT representations can replace the hand-engineered linguistic inputs, such as POS tags, dependency parses and…"
authors: ["Peng Shi", "Jimmy Lin"]
year: 2019
venue: "arXiv:1904.05255"
source: "https://arxiv.org/abs/1904.05255"
doi: "10.48550/arxiv.1904.05255"
arxiv: "1904.05255"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2019-simple-bert-models-relation-extraction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2019-simple-bert-models-relation-extraction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Simple BERT Models for Relation Extraction and Semantic Role Labeling

# Simple BERT Models for Relation Extraction and Semantic Role Labeling

**Authors:** [Peng Shi](https://md.research.elamur.ai/free/en/authors/peng-shi/), [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2019 · **Venue:** arXiv:1904.05255 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.05255) · [PDF](https://arxiv.org/pdf/1904.05255) · [DOI 10.48550/arxiv.1904.05255](https://doi.org/10.48550/arxiv.1904.05255) · [arXiv 1904.05255](https://arxiv.org/abs/1904.05255) · [OpenAlex](https://openalex.org/W2936215830)

**Citations:** 170 citations (openalex-snapshot · 2026-10-14) · [170](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Shi and Lin examine whether pretrained BERT representations can replace the hand-engineered linguistic inputs, such as POS tags, dependency parses and decoding constraints, that leading relation extraction and semantic role labeling (SRL) systems relied on. Such features depend on parsers, which may not exist for a given language or may break down on out-of-domain text.

For relation extraction, the sentence is paired with its subject and object, and entity mentions are replaced by typed masks. BERT outputs are combined with relative-position embeddings and passed through a BiLSTM and an MLP.

For SRL, the predicate is appended to the input. A predicate-indicator embedding, a BiLSTM and an MLP tag BIO argument labels, and a separate MLP labeler handles predicate sense disambiguation.

Experiments on TACRED, CoNLL 2005, CoNLL 2009 and CoNLL 2012 show these feature-free models matching or beating prior single-model results, and on some benchmarks they also beat ensembles. The work establishes strong, simple baselines and suggests that language pretraining can reduce reliance on syntactic tooling. This is relevant for languages with weaker parsing resources.

## Key points
- **TACRED:** BERT-LSTM-base reaches F1 67.8 (P 73.3, R 63.1). This is the best reported single-model score. It remains slightly below the Zhang et al. (2018) ensemble (68.2) because of lower recall.
- **CoNLL 2009 predicate sense disambiguation:** a plain BERT-base MLP scores 96.32 on dev, 96.88 on test and 90.63 on Brown, beating the feature-based He et al. (2018b).
- **CoNLL 2009 dependency-based SRL (end-to-end):** BERT-LSTM-large gets F1 92.4 in-domain and 85.7 on Brown. The previous best, Li et al. (2019), had 90.4 and 81.5, and existing ensembles are also surpassed.
- **Span-based SRL:** BERT-LSTM-large scores F1 88.8 on CoNLL 2005, 82.0 on Brown and 86.5 on CoNLL 2012. It outperforms the Ouchi et al. (2018) ensemble except on CoNLL 2012 (87.0), where constrained decoding gives that ensemble higher precision.
- **Setup:** the models use BERT base-cased and large-cased, an LSTM hidden size of 768, an MLP hidden size of 300 and a learning rate of 5×10⁻⁵. No syntactic features or declarative constraints are used.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Semantic role labeling](https://md.research.elamur.ai/free/en/topics/semantic-role-labeling/)

## Cited by

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 158 not yet.

- [Large language models are few-shot clinical information extractors](https://md.research.elamur.ai/free/en/papers/agrawal-2022-large-language-models-are-few/)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/)
- [Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora](https://md.research.elamur.ai/free/en/papers/bugert-2021-generalizing-cross-document-event-coreference/)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/)
- [Incorporating Syntax and Semantics in Coreference Resolution with Heterogeneous Graph Attention Network](https://md.research.elamur.ai/free/en/papers/jiang-2021-incorporating-syntax-semantics-coreference-resolution/)
- [QANom: Question-Answer driven SRL for Nominalizations](https://md.research.elamur.ai/free/en/papers/klein-2020-qanom-question-answer-driven-srl/)
- [Document-Level Event Argument Extraction by Conditional Generation](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities](https://md.research.elamur.ai/free/en/papers/orlando-2023-exploring-non-verbal-predicates-semantic/)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/)
- [Pairwise Representation Learning for Event Coreference](https://md.research.elamur.ai/free/en/papers/yu-2022-pairwise-representation-learning-event-coreference/)
- [Reasoning Over Semantic-Level Graph for Fact Checking](https://md.research.elamur.ai/free/en/papers/zhong-2020-reasoning-over-semantic-level-graph/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{shi2019simple,
  title = {{Simple BERT Models for Relation Extraction and Semantic Role Labeling}},
  author = {Shi, Peng and Lin, Jimmy},
  year = {2019},
  doi = {10.48550/arxiv.1904.05255},
  eprint = {1904.05255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.05255}
}
```

- [← previous Similarity of Neural Network Representations Revisited](https://md.research.elamur.ai/free/en/papers/kornblith-2019-similarity-neural-network-representations-revisited/)
- [next → Simple, Scalable Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Improving relation extraction by pre-trained language representations. In AKBC. [↗](https://doi.org/10.24432/c5kw2w)
2. Xavier Carreras and Lluı́s Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004. [find](https://scholar.google.com/scholar?q=Xavier+Carreras+and+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2E+2004%2E+Introduction+to+the+CoNLL%2D2004+shared+task%3A+Semantic+role+labeling%2E+In+Proceedings+of+the+Eighth+Conference+on+Computational+Natural+Language+Learning+%28CoNLL)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
4. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. [find](https://scholar.google.com/scholar?q=Anthony+Fader%2C+Stephen+Soderland%2C+and+Oren+Etzioni%2E+2011%2E+Identifying+relations+for+open+information+extraction%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
5. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–18. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2C+Massimiliano+Ciaramita%2C+Richard+Johansson%2C+Daisuke+Kawahara%2C+Maria+Anto%CC%80nia+Mart%C4%B1%CC%81%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Adam+Meyers%2C+Joakim+Nivre%2C+Sebastian+Pado%CC%81%2C+Jan+S%CC%8Cte%CC%8Cpa%CC%81nek%2C+et+al%2E+2009%2E+The+CoNLL%2D200) [our card](https://md.research.elamur.ai/free/en/papers/hajic-2009-conll-2009-shared-task-syntactic/)
6. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018a. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364–369. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Kenton+Lee%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2018a%2E+Jointly+predicting+predicates+and+arguments+in+neural+semantic+role+labeling%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Associatio) [our card](https://md.research.elamur.ai/free/en/papers/he-2018-jointly-predicting-predicates-arguments-neural/)
7. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473–483. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Kenton+Lee%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2017%2E+Deep+semantic+role+labeling%3A+What+works+and+what%E2%80%99s+next%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+L) [our card](https://md.research.elamur.ai/free/en/papers/he-2017-deep-semantic-role-labeling-what/)
8. Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018b. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2061–2071. [find](https://scholar.google.com/scholar?q=Shexia+He%2C+Zuchao+Li%2C+Hai+Zhao%2C+and+Hongxiao+Bai%2E+2018b%2E+Syntax+for+semantic+role+labeling%2C+to+be%2C+or+not+to+be%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguist)
9. Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1609.02907) [our card](https://md.research.elamur.ai/free/en/papers/kipf-2016-semi-supervised-classification-graph-convolutional/)
10. Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401–2411. [find](https://scholar.google.com/scholar?q=Zuchao+Li%2C+Shexia+He%2C+Jiaxun+Cai%2C+Zhuosheng+Zhang%2C+Hai+Zhao%2C+Gongshen+Liu%2C+Linlin+Li%2C+and+Luo+Si%2E+2018%2E+A+unified+syntax%2Daware+framework+for+semantic+role+labeling%2E+In+Proceedings+of+the+2018+Conferen)
11. Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Zuchao+Li%2C+Shexia+He%2C+Hai+Zhao%2C+Yiqing+Zhang%2C+Zhuosheng+Zhang%2C+Xi+Zhou%2C+and+Xiang+Zhou%2E+2019%2E+Dependency+or+span%2C+end%2Dto%2Dend+uniform+semantic+role+labeling%2E+In+Proceedings+of+the+33rd+AAAI+Conference+)
12. Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 411–420. [find](https://scholar.google.com/scholar?q=Diego+Marcheggiani%2C+Anton+Frolov%2C+and+Ivan+Titov%2E+2017%2E+A+simple+and+accurate+syntax%2Dagnostic+neural+model+for+dependency%2Dbased+semantic+role+labeling%2E+In+Proceedings+of+the+21st+Conference+on+Computa) [our card](https://md.research.elamur.ai/free/en/papers/marcheggiani-2017-simple-accurate-syntax-agnostic-neural/)
13. Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506–1515. [find](https://scholar.google.com/scholar?q=Diego+Marcheggiani+and+Ivan+Titov%2E+2017%2E+Encoding+sentences+with+graph+convolutional+networks+for+semantic+role+labeling%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language) [our card](https://md.research.elamur.ai/free/en/papers/marcheggiani-2017-encoding-sentences-graph-convolutional-networks/)
14. Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1630–1642. [find](https://scholar.google.com/scholar?q=Hiroki+Ouchi%2C+Hiroyuki+Shindo%2C+and+Yuji+Matsumoto%2E+2018%2E+A+span+selection+model+for+semantic+role+labeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+p)
15. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
16. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143– 152. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Hwee+Tou+Ng%2C+Anders+Bjo%CC%88rkelund%2C+Olga+Uryupina%2C+Yuchen+Zhang%2C+and+Zhi+Zhong%2E+2013%2E+Towards+robust+linguistic+analysis+using+OntoNotes%2E+In+Proceedings)
17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
18. Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1192–1202. [↗](https://doi.org/10.48550/arxiv.1605.07515) [our card](https://md.research.elamur.ai/free/en/papers/roth-2016-neural-semantic-role-labeling-dependency/)
19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
20. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 12–21. [find](https://scholar.google.com/scholar?q=Dan+Shen+and+Mirella+Lapata%2E+2007%2E+Using+semantic+roles+to+improve+question+answering%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Computational) [our card](https://md.research.elamur.ai/free/en/papers/shen-2007-semantic-roles-improve-question-answering/)
21. Peng Shi and Yue Zhang. 2017. Joint bi-affine parsing and semantic role labeling. In Proceedings of the 2017 International Conference on Asian Language Processing (IALP), pages 338–341. [find](https://scholar.google.com/scholar?q=Peng+Shi+and+Yue+Zhang%2E+2017%2E+Joint+bi%2Daffine+parsing+and+semantic+role+labeling%2E+In+Proceedings+of+the+2017+International+Conference+on+Asian+Language+Processing+%28IALP%29%2C+pages+338%E2%80%93341%2E)
22. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Patrick+Verga%2C+Daniel+Andor%2C+David+Weiss%2C+and+Andrew+McCallum%2E+2018%2E+Linguistically%2Dinformed+self%2Dattention+for+semantic+role+labeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirica) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2018-linguistically-informed-self-attention-semantic/)
23. Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. 2019. Simplifying graph convolutional networks. arXiv:1902.07153. [↗](https://arxiv.org/abs/1902.07153)
24. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205–2215. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Peng+Qi%2C+and+Christopher+D%2E+Manning%2E+2018%2E+Graph+convolution+over+pruned+dependency+trees+improves+relation+extraction%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-graph-convolution-over-pruned-dependency/)
25. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+the+2017+Conference+on+Empir)