--- title: "Entity-Relation Extraction as Multi-Turn Question — Li 2019" description: "The paper addresses joint extraction of entities and relations from raw text." authors: ["Xiaoya Li", "Fan Yin", "Zijun Sun", "Xiayu Li", "Arianna Yuan", "Duo Chai", "Mingxin Zhou", "Jiwei Li"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1129/" doi: "10.18653/v1/p19-1129" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/li-2019-entity-relation-extraction-as-multi/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2019-entity-relation-extraction-as-multi/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Entity-Relation Extraction as Multi-Turn Question Answering # Entity-Relation Extraction as Multi-Turn Question Answering **Authors:** [Xiaoya Li](https://md.research.elamur.ai/free/en/authors/xiaoya-li/), Fan Yin, Zijun Sun, Xiayu Li, [Arianna Yuan](https://md.research.elamur.ai/free/en/authors/arianna-yuan/), Duo Chai, Mingxin Zhou, [Jiwei Li](https://md.research.elamur.ai/free/en/authors/jiwei-li/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1129/) · [PDF](https://aclanthology.org/P19-1129.pdf) · [DOI 10.18653/v1/p19-1129](https://doi.org/10.18653/v1/p19-1129) · [OpenAlex](https://openalex.org/W2949922292) **Citations:** 336 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses joint extraction of entities and relations from raw text. The authors argue that the usual triplet format, REL(e1, e2), cannot capture hierarchical dependencies between tags. For example, a person's position depends on the company, and a time period depends on that position. They reformulate the task as multi-turn question answering. Each entity and relation type gets a question template. A machine reading comprehension (MRC) model answers each question by predicting text spans, and a NONE answer means the relation is absent. Entities found in earlier turns are inserted into later questions, much like slot filling in dialogue systems. Reinforcement learning can be added on top of this sequential process. The approach sets new state-of-the-art results on ACE04, ACE05 and CoNLL04. The authors also release RESUME, a Chinese dataset of biographical texts that needs deeper, multi-step reasoning. This work matters because it shows that recasting information extraction as reading comprehension handles distant entity pairs and overlapping relations better than classic pipelined or joint models. ## Key points - Relation F1 on the benchmarks, as reported in the abstract: ACE04 49.4 (+1.0), ACE05 60.2 (+0.6), CoNLL04 68.9 (+2.1). These benchmarks need only two QA turns: head entities first, then relations together with tail entities. - RESUME (Chinese) requires four to five QA turns and covers Person, Company, Position and Time. Overall F1 is 92.1 for multi-turn QA and 92.5 with RL, compared with 90.8 for a BERT tagging+Biaffine dependency baseline and 89.8 for tagging+relation. - In ablations, natural-language questions consistently beat coarse pseudo-questions across all datasets. - On ACE05, joint training with loss weight λ=0.7 gave the best relation F1 (60.2). Training on head entities alone (λ=0) did not give the best entity F1, which suggests the relation stage helps entity extraction. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 101 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 87 not yet. - [Large language models are few-shot clinical information extractors](https://md.research.elamur.ai/free/en/papers/agrawal-2022-large-language-models-are-few/) - [Event Extraction by Answering (Almost) Natural Questions](https://md.research.elamur.ai/free/en/papers/du-2020-event-extraction-answering-almost-natural/) - [Bridging Anaphora Resolution as Question Answering](https://md.research.elamur.ai/free/en/papers/hou-2020-bridging-anaphora-resolution-as-question/) - [Not Just Classification: Recognizing Implicit Discourse Relation on Joint Modeling of Classification and Generation](https://md.research.elamur.ai/free/en/papers/jiang-2021-not-just-classification-recognizing-implicit/) - [Event Extraction as Multi-turn Question Answering](https://md.research.elamur.ai/free/en/papers/li-2020-event-extraction-as-multi-turn/) - [A Unified MRC Framework for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/li-2020-unified-mrc-framework-named-entity/) - [Event Extraction as Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/liu-2020-event-extraction-as-machine-reading/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking](https://md.research.elamur.ai/free/en/papers/wang-2020-tplinker-single-stage-joint-extraction/) - [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://md.research.elamur.ai/free/en/papers/wang-2020-two-are-better-than-one/) - [A Novel Cascade Binary Tagging Framework for Relational Triple Extraction](https://md.research.elamur.ai/free/en/papers/wei-2020-novel-cascade-binary-tagging-framework/) - [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://md.research.elamur.ai/free/en/papers/wei-2023-chatie-zero-shot-information-extraction/) - [CorefQA: Coreference Resolution as Query-based Span Prediction](https://md.research.elamur.ai/free/en/papers/wu-2020-corefqa-coreference-resolution-as-query/) - [A Frustratingly Easy Approach for Entity and Relation Extraction](https://md.research.elamur.ai/free/en/papers/zhong-2020-frustratingly-easy-approach-entity-relation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{li2019entity, title = {{Entity-Relation Extraction as Multi-Turn Question Answering}}, author = {Li, Xiaoya and Yin, Fan and Sun, Zijun and Li, Xiayu and Yuan, Arianna and Chai, Duo and Zhou, Mingxin and Li, Jiwei}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1129}, url = {https://doi.org/10.18653/v1/p19-1129} } ``` - [← previous Entity, Relation, and Event Extraction with Contextualized Span Representations](https://md.research.elamur.ai/free/en/papers/wadden-2019-entity-relation-event-extraction-contextualized/) - [next → EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) ## References 9 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2830–2836. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Giannis+Bekoulis%2C+Johannes+Deleu%2C+Thomas+Demeester%2C+and+Chris+Develder%2E+2018%2E+Adversarial+training+for+multi%2Dcontext+joint+entity+and+relation+extraction%2E+In+Proceedings+of+the+2018+Conference+on+Empi) 2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Je%CC%81ro%CC%82me+Louradour%2C+Ronan+Collobert%2C+and+Jason+Weston%2E+2009%2E+Curriculum+learning%2E+In+Proceedings+of+the+26th+annual+international+conference+on+machine+learning%2C+pages+41%E2%80%9348%2E+ACM%2E) [our card](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/) 3. Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 551– 560. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yee+Seng+Chan+and+Dan+Roth%2E+2011%2E+Exploiting+syntactico%2Dsemantic+structures+for+relation+extraction%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+L) 4. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on Models Li and Ji (2014) Miwa and Bansal (2016) Katiyar and Cardie (2017) Zhang et al. (2017) Sun et al. (2018) Multi-turn QA Entity P 85.2 82.9 84.0 -83.9 8… [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+Manning%2E+2014%2E+A+fast+and+accurate+dependency+parser+using+neural+networks%2E+In+Proceedings+of+the+2014+conference+on+Models+Li+and+Ji+%282014%29+Miwa+and+Bansal+%282016%29+Katiyar+a) [our card](https://md.research.elamur.ai/free/en/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/) 5. Entity R 76.9 83.9 81.3 -83.2 84.9 Entity F 80.8 83.4 82.6 83.5 83.6 84.8 Relation P 65.4 57.2 55.5 -64.9 64.8 Relation R 39.8 54.0 51.8 -55.1 56.2 Relation F 49.5 55.6 53.6 57.5 59.6 60.2 (+0.6) *Table 7: Results of different models on the ACE05 test set. Results for pipelined methods are omitted s… [find](https://scholar.google.com/scholar?q=Entity+R+76%2E9+83%2E9+81%2E3+%2D83%2E2+84%2E9+Entity+F+80%2E8+83%2E4+82%2E6+83%2E5+83%2E6+84%2E8+Relation+P+65%2E4+57%2E2+55%2E5+%2D64%2E9+64%2E8+Relation+R+39%2E8+54%2E0+51%2E8+%2D55%2E1+56%2E2+Relation+F+49%2E5+55%2E6+53%2E6+57%2E5+59%2E6+60%2E2+%28%2B0%2E6%29+%2ATab) 6. Entity R – – – 86.6 Entity F1 80.7 85.6 83.6 87.8 Relation P – – – 69.2 Relation R – – – 68.2 Relation F 61.0 67.8 62.0 68.9 (+2.1) *Table 8: Comparison of the proposed method with the previous models on the CoNLL04 dataset. Precision and recall values of baseline models were not reported in the pre… [find](https://scholar.google.com/scholar?q=Entity+R+%E2%80%93+%E2%80%93+%E2%80%93+86%2E6+Entity+F1+80%2E7+85%2E6+83%2E6+87%2E8+Relation+P+%E2%80%93+%E2%80%93+%E2%80%93+69%2E2+Relation+R+%E2%80%93+%E2%80%93+%E2%80%93+68%2E2+Relation+F+61%2E0+67%2E8+62%2E0+68%2E9+%28%2B2%2E1%29+%2ATable+8%3A+Comparison+of+the+proposed+method+with+the+previous+models) 7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E) 8. Timothy Dozat and Christopher D Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D+Manning%2E+2016%2E+Deep+biaffine+attention+for+neural+dependency+parsing%2E+arXiv+preprint+arXiv%3A1611%2E01734%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/) 9. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. *Table 9: Comparing of the effect of natural language questions with pseudo-questions.* EXAMPLE 1 [find](https://scholar.google.com/scholar?q=Matthew+Dunn%2C+Levent+Sagun%2C+Mike+Higgins%2C+V+Ugur+Guney%2C+Volkan+Cirik%2C+and+Kyunghyun+Cho%2E+2017%2E+Searchqa%3A+A+new+q%26a+dataset+augmented+with+context+from+a+search+engine%2E+arXiv+preprint+arXiv%3A1704%2E05179%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/)