--- title: "Semantically Annotated Snapshot of the English Wikipedia (2008)" description: "The release is distributed under the GNU FDL in a simple tabular format. 85.85% LAS. 9 references, cited by 7 papers here." authors: ["Jordi Atserias", "Hugo Zaragoza", "Massimiliano Ciaramita", "Giuseppe Attardi"] year: 2008 venue: "Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)" source: "https://aclanthology.org/L08-1165/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/atserias-2008-semantically-annotated-snapshot-english-wikipedia/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/atserias-2008-semantically-annotated-snapshot-english-wikipedia/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2008](https://md.research.elamur.ai/free/en/papers/2008/) / Semantically Annotated Snapshot of the English Wikipedia # Semantically Annotated Snapshot of the English Wikipedia **Authors:** Jordi Atserias, [Hugo Zaragoza](https://md.research.elamur.ai/free/en/authors/hugo-zaragoza/), [Massimiliano Ciaramita](https://md.research.elamur.ai/free/en/authors/massimiliano-ciaramita/) and [Giuseppe Attardi](https://md.research.elamur.ai/free/en/authors/giuseppe-attardi/) · **Year:** 2008 · **Venue:** [Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2008](https://md.research.elamur.ai/free/en/venues/lrec/2008/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L08-1165/) · [PDF](https://aclanthology.org/L08-1165.pdf) · [OpenAlex](https://openalex.org/W40305183) **Citations:** 45 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/atserias-2008-semantically-annotated-snapshot-english-wikipedia/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Wikipedia had become a common resource for NLP and IR research. But running a full linguistic pipeline over the whole encyclopedia took computing power many groups lacked. Teams also prepared different dumps in different ways, so their results were hard to compare. The authors address this by releasing SW1, a single fixed, pre-processed copy of the English Wikipedia (the 2006-11-04 dump, taken from the University of Amsterdam XML version) for shared use. Every article was stripped of markup, split into sentences and tokens, and then annotated with: - PoS tags - WordNet lemmas - DeSR dependency trees - three kinds of semantic labels (WordNet supersenses and two named-entity schemes) The original hyperlinks were aligned back onto the tokens. A companion file records which entities appear in which sentence, from which an entity–passage graph can be built. The release is distributed under the GNU FDL in a simple tabular format. The authors warn that quality falls on Wikipedia-specific material such as tables and lists. They present the corpus as a testbed for studying domain adaptation. ## Key points - **Scale:** 1,490,688 entries (redirects excluded), yielding 843,199,595 tokens in 74,924,392 sentences. The data is split into 3,000 UTF-8 files of roughly 500 entries each, in the "multitag" format. - **Syntax and lemmas:** - PoS tags come from the SuperSense Tagger trained on the WSJ Penn Treebank (97.1% on WSJ). - Lemmas come from WordNet's morph functions. - Dependencies come from DeSR, which scored 85.85% LAS and 86.99% UAS in the CoNLL 2007 English task. - **Semantic layers:** all three are built on the SuperSense Tagger, a first-order HMM trained with a regularized averaged perceptron. - WNSS: 45 WordNet supersenses, trained on Semcor, about 80% F1. - WSJ: BBN entity, nominal and numeric types, about 87% F1. - WSJCoNLL: BBN labels mapped by hand to PER/ORG/LOC/MISC, about 91% F1. - **Multitag format:** one token per line, with tab-separated columns for form, PoS, lemma, CoNLL, WNSS, WSJ, head, dependency label and link. Multi-token spans and links use IOB encoding, and `%%#` lines carry document and sentence metadata. - **Entity graph:** a sentence-to-entity file (WSJ tagset) defines a bipartite "entity containment" graph, in which an entity's degree equals its passage frequency. This graph and a hyperlink graph were built with fastutils, MG4J and WebGraph. ## Interesting ideas - Keeping all annotation layers side by side in one token-per-line table, with IOB marking for spans, makes it easy to compare layers and spot conflicts between them. - A second, coarser tagger was obtained by mapping a fine-grained entity tagset onto a smaller one with a hand-written table and retraining, rather than annotating new data. - After processing, the clean text was re-aligned with the original markup, which turned the source's hyperlinks into an extra token-level annotation column. - Linking every sentence to the entities it mentions forms a bipartite graph, and an entity's number of connections directly gives its passage frequency. - The authors point out that taggers and parsers degrade on tables, lists and references, which differ from the newswire they were trained on. They offer the corpus partly as a way to study this shift. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): crate en could export lemmas, tags, supersense-style classes and tree columns in one aligned dump, so a human can trace exactly where layers diverge. - **I2** (learn): a hand-made map from rich internal entity or role labels to standard coarse sets could produce extra comparable training and evaluation targets at almost no cost. - **I3** (world): wiki-style links recovered this way can act as weak, free evidence that a mention points to a specific entity, which helps seed the knowledge graph and entity linking. - **I4** (world): a mention-to-sentence index per text would let the world model ask cheaply which entities co-occur and how prominent each one is, in an explainable way. - **I5** (learn): evaluation should be reported separately by text region (prose vs. lists vs. tables), so that drops caused by domain shift are visible rather than averaged away. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of the Basque Country](https://md.research.elamur.ai/free/en/universities/university-of-the-basque-country/) — University, Leioa, Spain · 57 works · 16,520 citations - [University of Turin](https://md.research.elamur.ai/free/en/universities/university-of-turin/) — University, Turin, Italy · 33 works · 6,637 citations - [Yahoo (United Kingdom)](https://md.research.elamur.ai/free/en/companies/yahoo-united-kingdom/) — Company, London, United Kingdom · 3 works · 294 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works ## Cite Cite as: Jordi Atserias, Hugo Zaragoza, Massimiliano Ciaramita and Giuseppe Attardi. 2008. Semantically Annotated Snapshot of the English Wikipedia. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{atserias2008semantically, title = {{Semantically Annotated Snapshot of the English Wikipedia}}, author = {Jordi Atserias, Hugo Zaragoza, Massimiliano Ciaramita and Giuseppe Attardi}, booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)}, year = {2008}, doi = {10.63317/4uqppbbzs9c9}, url = {https://doi.org/10.63317/4uqppbbzs9c9} } ``` - [← previous Contradictions and Justifications: Extensions to the Textual Entailment Task](https://md.research.elamur.ai/free/en/papers/voorhees-2008-contradictions-justifications-extensions-textual-entailment/) - [next → Combining Word Sense and Usage for Modeling Frame Semantics](https://md.research.elamur.ai/free/en/papers/cao-2008-combining-word-sense-usage-modeling/) - [↑ back to papers of 2008](https://md.research.elamur.ai/free/en/papers/2008/) ## References 9 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. G. Attardi, F. Dell’Orletta, M. Simi, A. Chanev, and M. Ciaramita. 2007. Multilingual dependency parsing and domain adaptation using desr. In Proceedings the CoNLL Shared Task Session of EMNLP-CoNLL 2007. [find](https://scholar.google.com/scholar?q=G%2E+Attardi%2C+F%2E+Dell%E2%80%99Orletta%2C+M%2E+Simi%2C+A%2E+Chanev%2C+and+M%2E+Ciaramita%2E+2007%2E+Multilingual+dependency+parsing+and+domain+adaptation+using+desr%2E+In+Proceedings+the+CoNLL+Shared+Task+Session+of+EMNLP%2DCoNLL+2) 2. M. Ciaramita and Y. Altun. 2005. Named-entity recognition in novel domains with external lexical knowledge. In Advances in Structured Learning for Text and Speech Processing (NIPS 2005). [find](https://scholar.google.com/scholar?q=M%2E+Ciaramita+and+Y%2E+Altun%2E+2005%2E+Named%2Dentity+recognition+in+novel+domains+with+external+lexical+knowledge%2E+In+Advances+in+Structured+Learning+for+Text+and+Speech+Processing+%28NIPS+2005%29%2E) 3. M. Ciaramita and Y. Altun. 2006. [Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger](https://md.research.elamur.ai/free/en/papers/ciaramita-2006-broad-coverage-sense-disambiguation-information/). In Proceedings of the EMNLP. 4. E. Gabrilovich and S. Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Manuela M. Veloso, editor, International Joint Conference on Artificial Intelligence (IJ-CAI), pages 1606–1611. [find](https://scholar.google.com/scholar?q=E%2E+Gabrilovich+and+S%2E+Markovitch%2E+2007%2E+Computing+semantic+relatedness+using+wikipedia%2Dbased+explicit+semantic+analysis%2E+In+Manuela+M%2E+Veloso%2C+editor%2C+International+Joint+Conference+on+Artificial+Inte) 5. M. Marcus, G. Kim, M. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger. 1994. [The Penn treebank: Annotating predicate argument structure](https://md.research.elamur.ai/free/en/papers/marcus-1994-penn-treebank-annotating-predicate-argument/). 6. Rada Mihalcea. 2007. Using wikipedia for automatic word sense disambiguation. In Proceedings ofThe North American Chapter of the Association for Computational Linguistics(NAACL) HLT 2007, pages 196–203. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea%2E+2007%2E+Using+wikipedia+for+automatic+word+sense+disambiguation%2E+In+Proceedings+ofThe+North+American+Chapter+of+the+Association+for+Computational+Linguistics%28NAACL%29+HLT+2007%2C+pages+196%E2%80%9320) 7. G.A. Miller, C. Leacock, R. Tengi, and R.Bunker. 1993. [A semantic concordance](https://md.research.elamur.ai/free/en/papers/miller-1993-semantic-concordance/). In CA: Morgan Kauf-mann Publishers San Mateo, editor, Proceedings of the ARPA Human Language Technology Workshop., Princeton, NJ. 8. Erik F. Tjong Kim Sang and Fien De Muelder. 2003. [Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In CoNLL 2003 Shared Task, pages 142–147. [↗](https://doi.org/10.48550/arxiv.cs/0306050) 9. H. Zaragoza, H. Rode, P. Mika, J. Atserias, M. Ciaramita, and G. Attardi. 2007. Ranking very many typed entities on wikipedia. In CIKM, pages 1015–1018. [http://download.freebase.com/wex/http://medialab.di.unipi.it/wiki/index.php/Analisi](http://download.freebase.com/wex/http://medialab.di.unipi.it/wiki/index.php/Analisi) di testi per il Semantic Web e il Question Answering [↗](http://download.freebase.com/wex/http://medialab.di.unipi.it/wiki/index.php/Analisi) *Updated 1 November 2026*