---
title: "TACRED Revisited — Alt 2020"
description: "TACRED is a large, widely used crowdsourced benchmark for Relation Extraction (RE)."
authors: ["Christoph Alt", "Aleksandra Gabryszak", "Leonhard Hennig"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.142/"
doi: "10.18653/v1/2020.acl-main.142"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

# TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

**Authors:** Christoph Alt, Aleksandra Gabryszak, [Leonhard Hennig](https://md.research.elamur.ai/free/en/authors/leonhard-hennig/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.142/) · [PDF](https://aclanthology.org/2020.acl-main.142.pdf) · [DOI 10.18653/v1/2020.acl-main.142](https://doi.org/10.18653/v1/2020.acl-main.142) · [OpenAlex](https://openalex.org/W3023215888)

**Citations:** 132 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
TACRED is a large, widely used crowdsourced benchmark for Relation Extraction (RE). Even the strongest pre-trained models still get roughly 30% of it wrong. The authors ask whether this reflects a true performance ceiling, or whether noisy crowd labels and dataset properties are to blame. They ranked development and test instances by how many of 49 RE models misclassified them. Trained linguists then re-annotated about 5k of the hardest cases, plus a control sample, following the TAC KBP guidelines. Roughly half of the difficult examples turned out to carry wrong labels, and fixing them raises the mean F1 of the baseline models by about 8 points. The authors then grouped the remaining model errors into linguistically motivated categories. They tested these error hypotheses on three state-of-the-art systems by automatically rewriting test inputs. Most residual errors come from two clusters of ambiguous relations. Models that see unmasked entities tend to rely on shallow cues. The work supplies a corrected evaluation set and argues that scoring should account for instance difficulty.

## Key points
- **TACRED** has about 106k sentence-level examples covering 41 relation types plus no_relation, labeled via Amazon Mechanical Turk. Around 79.5% of examples are negative.
- **Label noise** in the challenging subsets was substantial:
  - 49.9% of Test and 52.1% of Dev examples were revised.
  - Only about 9% of the easy control examples needed changes.
  - Re-annotator agreement was high (κ = 0.87 Test, 0.80 Dev), but agreement with the original crowd labels was only κ ≈ 0.55.
- **Effect on scores**: label errors cost about 8 F1 points on the test set. The average baseline F1 moves from 62.1 to 70.1 on the revised test set; KnowBERT moves from 71.5 to 79.3 and SpanBERT from 70.8 to 78.0.
- **Error analysis**:
  - Model mistakes were sorted into 9 RE error categories.
  - Rewriting experiments showed that KnowBERT, which sees unmasked entities, depends more on entity content.
  - SpanBERT, which uses entity masking, relies more on context and scores best under difficulty-weighted evaluation (61.9 vs. 58.7 F1).
- **Resource**: the revised labels are released as a patch to the original LDC data at github.com/DFKI-NLP/tacrev.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 76 not yet.

- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://md.research.elamur.ai/free/en/papers/cl-2023-klie-annotation-error-detection-analyzing-past/)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://md.research.elamur.ai/free/en/papers/genzel-2024-varierr-nli-separating-annotation-error/)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://md.research.elamur.ai/free/en/papers/ma-2023-large-language-model-is-not/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{alt2020tacred,
  title = {{TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task}},
  author = {Alt, Christoph and Gabryszak, Aleksandra and Hennig, Leonhard},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.142},
  url = {https://doi.org/10.18653/v1/2020.acl-main.142}
}
```

- [← previous TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/)
- [next → Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation](https://md.research.elamur.ai/free/en/papers/gotou-2020-taking-correction-difficulty-into-account/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Improving relation extraction by pre-trained language representations. In Proceedings of the 2019 Conference on Automated Knowledge Base Construction, Amherst, Massachusetts. [find](https://scholar.google.com/scholar?q=Christoph+Alt%2C+Marc+Hu%CC%88bner%2C+and+Leonhard+Hennig%2E+2019%2E+Improving+relation+extraction+by+pre%2Dtrained+language+representations%2E+In+Proceedings+of+the+2019+Conference+on+Automated+Knowledge+Base+Constru)
2. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895–2905, Florence, Italy. Association for Comp… [find](https://scholar.google.com/scholar?q=Livio+Baldini+Soares%2C+Nicholas+FitzGerald%2C+Jeffrey+Ling%2C+and+Tom+Kwiatkowski%2E+2019%2E+Matching+the+blanks%3A+Distributional+similarity+for+relation+learning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+t)
3. Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12–23, Florence, Italy. Association for Computational… [find](https://scholar.google.com/scholar?q=Jeremy+Barnes%2C+Lilja+%C3%98vrelid%2C+and+Erik+Velldal%2E+2019%2E+Sentiment+analysis+is+not+solved%21+assessing+and+probing+sentiment+classification%2E+In+Proceedings+of+the+2019+ACL+Workshop+BlackboxNLP%3A+Analyzing+a)
4. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358–2367, Berlin, Germany. Association for C… [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Jason+Bolton%2C+and+Christopher+D%2E+Manning%2E+2016%2E+A+thorough+examination+of+the+CNN%2Fdaily+mail+reading+comprehension+task%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)
5. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+German+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+%24%26%21%5C%23%2A+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In+Pr) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
7. David Harbecke, Robert Schwarzenberg, and Christoph Alt. 2018. Learning explanations from language data. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 316–318, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Harbecke%2C+Robert+Schwarzenberg%2C+and+Christoph+Alt%2E+2018%2E+Learning+explanations+from+language+data%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+Neural+Networ)
8. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International… [find](https://scholar.google.com/scholar?q=Iris+Hendrickx%2C+Su+Nam+Kim%2C+Zornitsa+Kozareva%2C+Preslav+Nakov%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Sebastian+Pado%CC%81%2C+Marco+Pennacchiotti%2C+Lorenza+Romano%2C+and+Stan+Szpakowicz%2E+2010%2E+SemEval%2D2010+task+8%3A+Multi%2Dway+cla) [our card](https://md.research.elamur.ai/free/en/papers/hendrickx-2010-semeval-2010-task-8-multi/)
9. Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 1148–1158, Stroudsburg, PA, USA. Association for Compu… [find](https://scholar.google.com/scholar?q=Heng+Ji+and+Ralph+Grishman%2E+2011%2E+Knowledge+base+population%3A+Successful+approaches+and+challenges%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Lan)
10. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+202) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
11. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. ArXiv, abs/1907.10529. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S%2E+Weld%2C+Luke+S%2E+Zettlemoyer%2C+and+Omer+Levy%2E+2019%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+ArXiv%2C+abs%2F1907%2E10529%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
12. Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conferenc… [find](https://scholar.google.com/scholar?q=Najoung+Kim%2C+Roma+Patel%2C+Adam+Poliak%2C+Patrick+Xia%2C+Alex+Wang%2C+Tom+McCoy%2C+Ian+Tenney%2C+Alexis+Ross%2C+Tal+Linzen%2C+Benjamin+Van+Durme%2C+Samuel+R%2E+Bowman%2C+and+Ellie+Pavlick%2E+2019%2E+Probing+what+different+NLP+) [our card](https://md.research.elamur.ai/free/en/papers/kim-2019-probing-what-different-nlp-tasks/)
13. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Tom+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+wrong+reasons%3A+Diagnosing+syntactic+heuristics+in+natural+language+inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
14. Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 39–48, Denver, Colorado. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thien+Huu+Nguyen+and+Ralph+Grishman%2E+2015%2E+Relation+extraction%3A+Perspective+from+convolutional+neural+networks%2E+In+Proceedings+of+the+1st+Workshop+on+Vector+Space+Modeling+for+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2015-relation-extraction-perspective-convolutional-neural/)
15. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Timothy+Niven+and+Hung%2DYu+Kao%2E+2019%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pag)
16. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Process)
17. Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on… [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Robert+Logan%2C+Roy+Schwartz%2C+Vidur+Joshi%2C+Sameer+Singh%2C+and+Noah+A%2E+Smith%2E+2019%2E+Knowledge+enhanced+contextual+word+representations%2E+In+Proceedings+of+the+2019+Conferen)
18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. available as a preprint. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+available+as+a+preprint%2E)
19. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for C… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
20. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling Relations and Their Mentions without Labeled Text. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD ’10). [find](https://scholar.google.com/scholar?q=Sebastian+Riedel%2C+Limin+Yao%2C+and+Andrew+McCallum%2E+2010%2E+Modeling+Relations+and+Their+Mentions+without+Labeled+Text%2E+In+Proceedings+of+the+European+Conference+on+Machine+Learning+and+Knowledge+Discover)
21. Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. ArXiv, abs/1904.05255. [find](https://scholar.google.com/scholar?q=Peng+Shi+and+Jimmy+Lin%2E+2019%2E+Simple+bert+models+for+relation+extraction+and+semantic+role+labeling%2E+ArXiv%2C+abs%2F1904%2E05255%2E) [our card](https://md.research.elamur.ai/free/en/papers/shi-2019-simple-bert-models-relation-extraction/)
22. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 747–763, Florence, Italy. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Tongshuang+Wu%2C+Marco+Tulio+Ribeiro%2C+Jeffrey+Heer%2C+and+Daniel+Weld%2E+2019%2E+Errudite%3A+Scalable%2C+reproducible%2C+and+testable+error+analysis%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for)
23. Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016. Question answering on freebase via relation extraction and textual evidence. [find](https://scholar.google.com/scholar?q=Kun+Xu%2C+Siva+Reddy%2C+Yansong+Feng%2C+Songfang+Huang%2C+and+Dongyan+Zhao%2E+2016%2E+Question+answering+on+freebase+via+relation+extraction+and+textual+evidence%2E) [our card](https://md.research.elamur.ai/free/en/papers/xu-2016-question-answering-freebase-relation-extraction/)
24. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2326–2336. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+2326%E2%80%932336%2E+Association+for+Computational+Linguistics%2E)
25. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335–2344, Dublin, Ireland. Dublin City Universi… [find](https://scholar.google.com/scholar?q=Daojian+Zeng%2C+Kang+Liu%2C+Siwei+Lai%2C+Guangyou+Zhou%2C+and+Jun+Zhao%2E+2014%2E+Relation+classification+via+convolutional+deep+neural+network%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2014-relation-classification-convolutional-deep-neural/)
26. Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205–2215, Brussels, Belgium. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Peng+Qi%2C+and+Christopher+D%2E+Manning%2E+2018%2E+Graph+convolution+over+pruned+dependency+trees+improves+relation+extraction%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-graph-convolution-over-pruned-dependency/)
27. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45, Copenhagen, Denmark. Association for Compu… [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Victor+Zhong%2C+Danqi+Chen%2C+Gabor+Angeli%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Position%2Daware+attention+and+supervised+data+improve+slot+filling%2E+In+Proceedings+of+the+2017+Conference+on+Empir)
28. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451, Florence, Italy. Association for Computation… [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+ERNIE%3A+Enhanced+language+representation+with+informative+entities%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Asso)
29. Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Luisa+M%2E+Zintgraf%2C+Taco+S%2E+Cohen%2C+Tameem+Adel%2C+and+Max+Welling%2E+2017%2E+Visualizing+deep+neural+network+decisions%3A+Prediction+difference+analysis%2E+International+Conference+on+Learning+Representations%2E)