---
title: "REL: An Entity Linker Standing on the Shoulders of Giants (2020)"
description: "The authors argue that existing open-source linkers have several weaknesses. 28 references, cited by 46 papers here."
authors: ["Johannes M. van Hulst", "Faegheh Hasibi", "Koen Dercksen", "Krisztian Balog", "Arjen P. de Vries"]
year: 2020
venue: "SIGIR 2020 (demo)"
source: "https://arxiv.org/abs/2006.01969"
doi: "10.1145/3397271.3401416"
arxiv: "2006.01969"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hulst-2020-rel-entity-linker-standing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hulst-2020-rel-entity-linker-standing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / REL: An Entity Linker Standing on the Shoulders of Giants

# REL: An Entity Linker Standing on the Shoulders of Giants

**Authors:** Johannes M. van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog and Arjen P. de Vries · **Year:** 2020 · **Venue:** [SIGIR 2020 (demo)](https://md.research.elamur.ai/free/en/venues/international-acm-sigir-conference-on-research-and-development-in-information/) ([SIGIR 2020](https://md.research.elamur.ai/free/en/venues/international-acm-sigir-conference-on-research-and-development-in-information/2020/)) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3397271.3401416) · [PDF](https://arxiv.org/pdf/2006.01969) · [DOI 10.1145/3397271.3401416](https://doi.org/10.1145/3397271.3401416) · [arXiv 2006.01969](https://arxiv.org/abs/2006.01969) · [OpenAlex](https://openalex.org/W3033919759)

**Citations:** 62 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/hulst-2020-rel-entity-linker-standing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Entity linking (EL), which means spotting entity mentions in text and mapping them to knowledge-base identifiers, is often handled by third-party tools in retrieval pipelines. The authors argue that existing open-source linkers have several weaknesses. Some are no longer maintained, some rely on external services, some are slow, and many are tied to old Wikipedia snapshots. None of them use recent neural NLP methods. REL (Radboud Entity Linker) addresses this by combining existing state-of-the-art components in a modular pipeline:

- **Mention detection:** Flair NER.
- **Candidate selection:** a mention-entity prior plus Wikipedia2Vec context similarity.
- **Disambiguation:** the Ment-norm model of Le and Titov.

REL is trained on a 2019-07 Wikipedia dump and ships with scripts for rebuilding it on newer dumps. It runs on an ordinary laptop without a GPU and is released as a Python package and a REST API. On GERBIL benchmarks, REL clearly beats DBpedia Spotlight and WAT. In end-to-end EL it matches or exceeds the SOTA NLP system on about half of the datasets, though that system remains stronger for disambiguation-only evaluation. This gives IR researchers a current, replaceable-by-parts linker.

## Key points
- **Candidate selection:** up to 7 candidates per mention. Four come from the P(e|m) prior, built from Wikipedia and CrossWikis hyperlink counts plus YAGO. Three come from context similarity using Wikipedia2Vec embeddings.
- **Disambiguation model:** trained on AIDA-train and validated on AIDA-A, using GloVe embeddings. A logistic fit over the final scores yields confidence values.
- **Reproducibility check:** on AIDA-B, local ED micro F1 is 92.8 for REL (2014), 89.4 for REL (2019), and 93.1 for the original MulRel-NEL. The drop in the 2019 version is attributed to lower candidate recall with Wikipedia2Vec embeddings.
- **Efficiency:**
  - About 700 ms per ~300-word document.
  - Memory use can be kept to about 1.8GB by storing embeddings and the P(e|m) index in SQLite.
  - Mention detection dominates runtime: 0.44 s with a GPU versus 2.41 s on CPU. Disambiguation takes about 0.2 s either way.
- **Availability:** MIT-licensed. Components such as the NER tagger (e.g., spaCy instead of Flair) or the corpora behind the prior can be swapped.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Radboud University Nijmegen](https://md.research.elamur.ai/free/en/universities/radboud-university-nijmegen/) — University, Nijmegen, Netherlands · 21 works · 1,796 citations
- [University of Stavanger](https://md.research.elamur.ai/free/en/universities/university-of-stavanger/) — University, Stavanger, Norway · 2 works · 112 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

## Cite

Cite as: Johannes M. van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog and Arjen P. de Vries. 2020. REL: An Entity Linker Standing on the Shoulders of Giants. SIGIR 2020 (demo).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hulst2020rel,
  title = {{REL: An Entity Linker Standing on the Shoulders of Giants}},
  author = {Johannes M. {van Hulst}, Faegheh Hasibi, Koen Dercksen, Krisztian Balog and Arjen P. {de Vries}},
  booktitle = {SIGIR 2020 (demo)},
  year = {2020},
  doi = {10.1145/3397271.3401416},
  eprint = {2006.01969},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3397271.3401416}
}
```

- [← previous AMR-To-Text Generation with Graph Transformer](https://md.research.elamur.ai/free/en/papers/wang-2020-amr-text-generation-graph-transformer/)
- [next → An Empirical Study of Pre-trained Transformers for Arabic Information Extraction](https://md.research.elamur.ai/free/en/papers/lan-2020-empirical-study-pre-trained-transformers/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. [Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/). In Proc. of COLING ’18. 1638–1649.
2. Krisztian Balog. 2018. Entity-Oriented Search. The Information Retrieval Series, Vol. 39. Springer. [find](https://scholar.google.com/scholar?q=Krisztian+Balog%2E+2018%2E+Entity%2DOriented+Search%2E+The+Information+Retrieval+Series%2C+Vol%2E+39%2E+Springer%2E)
3. Krisztian Balog, Heri Ramampiaro, Naimdjon Takhirov, and Kjetil Nørvåg. 2013. Multi-step Classification Approaches to Cumulative Citation Recommendation. In Proc. of OAIR ’13. 121–128. [find](https://scholar.google.com/scholar?q=Krisztian+Balog%2C+Heri+Ramampiaro%2C+Naimdjon+Takhirov%2C+and+Kjetil+N%C3%B8rv%C3%A5g%2E+2013%2E+Multi%2Dstep+Classification+Approaches+to+Cumulative+Citation+Recommendation%2E+In+Proc%2E+of+OAIR+%E2%80%9913%2E+121%E2%80%93128%2E)
4. Razvan Bunescu and Marius Paşca. 2006. [Using Encyclopedic Knowledge for Named Entity Disambiguation](https://md.research.elamur.ai/free/en/papers/bunescu-2006-encyclopedic-knowledge-named-entity-disambiguation/). In Proc. of EACL ’06. 9–16.
5. Marco Cornolti, Paolo Ferragina, Massimiliano Ciaramita, Stefan Rüd, and Hinrich Schütze. 2018. SMAPH: A Piggyback Approach for Entity-Linking in Web Queries. ACM Trans. Inf. Syst. 37, 1 (2018). [find](https://scholar.google.com/scholar?q=Marco+Cornolti%2C+Paolo+Ferragina%2C+Massimiliano+Ciaramita%2C+Stefan+R%C3%BCd%2C+and+Hinrich+Sch%C3%BCtze%2E+2018%2E+SMAPH%3A+A+Piggyback+Approach+for+Entity%2DLinking+in+Web+Queries%2E+ACM+Trans%2E+Inf%2E+Syst%2E+37%2C+1+%282018%29%2E)
6. Silviu Cucerzan. 2007. Large-Scale Named Entity Disambiguation Based on Wikipedia Data. In Proc. of EMNLP-CoNLL ’07. 708–716. [find](https://scholar.google.com/scholar?q=Silviu+Cucerzan%2E+2007%2E+Large%2DScale+Named+Entity+Disambiguation+Based+on+Wikipedia+Data%2E+In+Proc%2E+of+EMNLP%2DCoNLL+%E2%80%9907%2E+708%E2%80%93716%2E)
7. Paolo Ferragina and Ugo Scaiella. 2010. [TAGME: On-the-fly Annotation of Short Text Fragments (by Wikipedia Entities)](https://md.research.elamur.ai/free/en/papers/ferragina-2010-tagme-fly-annotation-short/). In Proc. of CIKM ’10. 1625–1628.
8. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep Joint Entity Disambiguation with Local Neural Attention. In Proc. of EMNLP ’17. 2619–2629. [find](https://scholar.google.com/scholar?q=Octavian%2DEugen+Ganea+and+Thomas+Hofmann%2E+2017%2E+Deep+Joint+Entity+Disambiguation+with+Local+Neural+Attention%2E+In+Proc%2E+of+EMNLP+%E2%80%9917%2E+2619%E2%80%932629%2E)
9. Emma Gerritse, Faegheh Hasibi, and Arjen P. de Vries. Graph-Embedding Empowered Entity Retrieval. In Proc. of ECIR ’20. 97–110. [find](https://scholar.google.com/scholar?q=Emma+Gerritse%2C+Faegheh+Hasibi%2C+and+Arjen+P%2E+de+Vries%2E+Graph%2DEmbedding+Empowered+Entity+Retrieval%2E+In+Proc%2E+of+ECIR+%E2%80%9920%2E+97%E2%80%93110%2E)
10. Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. 2015. Entity Linking in Queries: Tasks and Evaluation. In Proc. of ICTIR ’15. 171–180. [find](https://scholar.google.com/scholar?q=Faegheh+Hasibi%2C+Krisztian+Balog%2C+and+Svein+Erik+Bratsberg%2E+2015%2E+Entity+Linking+in+Queries%3A+Tasks+and+Evaluation%2E+In+Proc%2E+of+ICTIR+%E2%80%9915%2E+171%E2%80%93180%2E)
11. Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. 2016. Exploiting Entity Linking in Queries for Entity Retrieval. In Proc. of ICTIR ’16. 209–218. [find](https://scholar.google.com/scholar?q=Faegheh+Hasibi%2C+Krisztian+Balog%2C+and+Svein+Erik+Bratsberg%2E+2016%2E+Exploiting+Entity+Linking+in+Queries+for+Entity+Retrieval%2E+In+Proc%2E+of+ICTIR+%E2%80%9916%2E+209%E2%80%93218%2E)
12. Faegheh Hasibi, Krisztian Balog, Darío Garigliotti, and Shuo Zhang. 2017. Nordlys: A Toolkit for Entity-Oriented and Semantic Search. In Proc. of SIGIR ’17. 1289–1292. [find](https://scholar.google.com/scholar?q=Faegheh+Hasibi%2C+Krisztian+Balog%2C+Dar%C3%ADo+Garigliotti%2C+and+Shuo+Zhang%2E+2017%2E+Nordlys%3A+A+Toolkit+for+Entity%2DOriented+and+Semantic+Search%2E+In+Proc%2E+of+SIGIR+%E2%80%9917%2E+1289%E2%80%931292%2E)
13. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Proc. of EMNLP ’11. 782–792. [find](https://scholar.google.com/scholar?q=Johannes+Hoffart%2C+Mohamed+Amir+Yosef%2C+Ilaria+Bordino%2C+Hagen+F%C3%BCrstenau%2C+Manfred+Pinkal%2C+Marc+Spaniol%2C+Bilyana+Taneva%2C+Stefan+Thater%2C+and+Gerhard+Weikum%2E+2011%2E+Robust+Disambiguation+of+Named+Entities+in)
14. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-End Neural Entity Linking. In Proc. of CoNLL ’18. 519–529. [find](https://scholar.google.com/scholar?q=Nikolaos+Kolitsas%2C+Octavian%2DEugen+Ganea%2C+and+Thomas+Hofmann%2E+2018%2E+End%2Dto%2DEnd+Neural+Entity+Linking%2E+In+Proc%2E+of+CoNLL+%E2%80%9918%2E+519%E2%80%93529%2E)
15. Phong Le and Ivan Titov. 2018. Improving Entity Linking by Modeling Latent Relations between Mentions. In Proc. of ACL ’18. 1595–1604. [find](https://scholar.google.com/scholar?q=Phong+Le+and+Ivan+Titov%2E+2018%2E+Improving+Entity+Linking+by+Modeling+Latent+Relations+between+Mentions%2E+In+Proc%2E+of+ACL+%E2%80%9918%2E+1595%E2%80%931604%2E)
16. Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. 2011. DBpedia Spotlight: Shedding Light on the Web of Documents. In Proc. of I-Semantics ’11. 1–8. [find](https://scholar.google.com/scholar?q=Pablo+N+Mendes%2C+Max+Jakob%2C+Andr%C3%A9s+Garc%C3%ADa%2DSilva%2C+and+Christian+Bizer%2E+2011%2E+DBpedia+Spotlight%3A+Shedding+Light+on+the+Web+of+Documents%2E+In+Proc%2E+of+I%2DSemantics+%E2%80%9911%2E+1%E2%80%938%2E)
17. Rada Mihalcea and Andras Csomai. 2007. Wikify! - Linking Documents to Encyclopedic Knowledge. In Proc. of CIKM ’07. 233–242. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Andras+Csomai%2E+2007%2E+Wikify%21+%2D+Linking+Documents+to+Encyclopedic+Knowledge%2E+In+Proc%2E+of+CIKM+%E2%80%9907%2E+233%E2%80%93242%2E)
18. David Milne and Ian H Witten. 2008. Learning to Link with Wikipedia. In Proc. of CIKM ’08. 509–518. [find](https://scholar.google.com/scholar?q=David+Milne+and+Ian+H+Witten%2E+2008%2E+Learning+to+Link+with+Wikipedia%2E+In+Proc%2E+of+CIKM+%E2%80%9908%2E+509%E2%80%93518%2E)
19. Aasish Pappu, Roi Blanco, Yashar Mehdad, Amanda Stent, and Kapil Thadani. 2017. Lightweight Multilingual Entity Extraction and Linking. In Proc. of WSDM ’17. 365–374. [find](https://scholar.google.com/scholar?q=Aasish+Pappu%2C+Roi+Blanco%2C+Yashar+Mehdad%2C+Amanda+Stent%2C+and+Kapil+Thadani%2E+2017%2E+Lightweight+Multilingual+Entity+Extraction+and+Linking%2E+In+Proc%2E+of+WSDM+%E2%80%9917%2E+365%E2%80%93374%2E)
20. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [Glove: Global Vectors for Word Representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proc. of EMNLP ’14. 1532–1543.
21. Francesco Piccinno, Paolo Ferragina, and Dipartimento Informatica. 2014. From TagME to WAT : a new Entity Annotator Categories and Subject Descriptors. (2014), 55–62. [find](https://scholar.google.com/scholar?q=Francesco+Piccinno%2C+Paolo+Ferragina%2C+and+Dipartimento+Informatica%2E+2014%2E+From+TagME+to+WAT+%3A+a+new+Entity+Annotator+Categories+and+Subject+Descriptors%2E+%282014%29%2C+55%E2%80%9362%2E)
22. John Platt. 2000. Probabilities for SV Machines. In Advances in Large-Margin Classifiers. 61–73. [find](https://scholar.google.com/scholar?q=John+Platt%2E+2000%2E+Probabilities+for+SV+Machines%2E+In+Advances+in+Large%2DMargin+Classifiers%2E+61%E2%80%9373%2E)
23. Ridho Reinanda, Edgar Meij, and Maarten de Rijke. 2015. Mining, Ranking and Recommending Entity Aspects. In Proc. of SIGIR ’15. 263–272. [find](https://scholar.google.com/scholar?q=Ridho+Reinanda%2C+Edgar+Meij%2C+and+Maarten+de+Rijke%2E+2015%2E+Mining%2C+Ranking+and+Recommending+Entity+Aspects%2E+In+Proc%2E+of+SIGIR+%E2%80%9915%2E+263%E2%80%93272%2E)
24. Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. 2018. GERBIL–Benchmarking Named Entity Recognition and Linking Consistently. Semantic Web 9, 5 (2018), 605–625. [find](https://scholar.google.com/scholar?q=Michael+R%C3%B6der%2C+Ricardo+Usbeck%2C+and+Axel%2DCyrille+Ngonga+Ngomo%2E+2018%2E+GERBIL%E2%80%93Benchmarking+Named+Entity+Recognition+and+Linking+Consistently%2E+Semantic+Web+9%2C+5+%282018%29%2C+605%E2%80%93625%2E)
25. Valentin I. Spitkovsky and Angel X. Chang. 2012. A Cross-Lingual Dictionary for English Wikipedia Concepts. In Proc. of LREC’12. 3168–3175. [find](https://scholar.google.com/scholar?q=Valentin+I%2E+Spitkovsky+and+Angel+X%2E+Chang%2E+2012%2E+A+Cross%2DLingual+Dictionary+for+English+Wikipedia+Concepts%2E+In+Proc%2E+of+LREC%E2%80%9912%2E+3168%E2%80%933175%2E)
26. Chenyan Xiong, Jamie Callan, and Tie-Yan Liu. 2017. Word-Entity Duet Representations for Document Ranking. In Proc. of SIGIR ’17. 763–772. [find](https://scholar.google.com/scholar?q=Chenyan+Xiong%2C+Jamie+Callan%2C+and+Tie%2DYan+Liu%2E+2017%2E+Word%2DEntity+Duet+Representations+for+Document+Ranking%2E+In+Proc%2E+of+SIGIR+%E2%80%9917%2E+763%E2%80%93772%2E)
27. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2018. Wikipedia2Vec: An Optimized Tool for Learning Embeddings of Words and Entities from Wikipedia. arXiv preprint 1812.06280 (2018). — [arXiv:1812.06280](https://arxiv.org/abs/1812.06280) [find](https://scholar.google.com/scholar?q=Ikuya+Yamada%2C+Akari+Asai%2C+Hiroyuki+Shindo%2C+Hideaki+Takeda%2C+and+Yoshiyasu+Takefuji%2E+2018%2E+Wikipedia2Vec%3A+An+Optimized+Tool+for+Learning+Embeddings+of+Words+and+Entities+from+Wikipedia%2E+arXiv+preprint+1)
28. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. [Joint learning of the embedding of words and entities for named entity disambiguation](https://md.research.elamur.ai/free/en/papers/yamada-2016-joint-learning-embedding-words-entities/). In Proc of CoNLL ’16. 250–259.

*Updated 2 November 2026*