---
title: "OpenCeres — paper by Lockard et al. (2019)"
description: "The authors define OpenIE for such websites: recover (subject, predicate, object) triples in which the… 28 references, cited by 25 papers here."
authors: ["Colin Lockard", "Prashant Shiralkar", "Xin Luna Dong"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1309/"
doi: "10.18653/v1/n19-1309"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lockard-2019-openceres-when-open-information-extraction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lockard-2019-openceres-when-open-information-extraction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / OpenCeres: When Open Information Extraction Meets the Semi-Structured Web

# OpenCeres: When Open Information Extraction Meets the Semi-Structured Web

**Authors:** Colin Lockard, Prashant Shiralkar and [Xin Luna Dong](https://md.research.elamur.ai/free/en/authors/xin-dong/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1309/) · [PDF](https://aclanthology.org/N19-1309.pdf) · [DOI 10.18653/v1/n19-1309](https://doi.org/10.18653/v1/n19-1309) · [OpenAlex](https://openalex.org/W2955858358)

**Citations:** 24 citations (OpenAlex, 2026-10-14) · [25](https://md.research.elamur.ai/free/en/papers/lockard-2019-openceres-when-open-information-extraction/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most OpenIE research targets plain sentences or HTML tables. Semi-structured detail pages, such as IMDb-style templates, hold many facts but have been left out. On these pages, a relation label and its value may sit far apart in no fixed arrangement. The authors define OpenIE for such websites: recover (subject, predicate, object) triples in which the predicate is a string taken from the page itself.

They build OpenCeres on top of the ClosedIE system Ceres, using three steps:
- **Candidate generation:** a cheap search pairs each possible object with nearby text fields that occur more often across the site.
- **Distant supervision:** a seed KB marks pairs whose relations already exist in the ontology.
- **Label propagation:** semi-supervised propagation over visual and DOM similarity finds similarly formatted pairs, which supplies training labels for new relations.

A classifier is then trained on these labels. To evaluate, they extend the SWDE corpus with dense ground-truth annotations. OpenCeres reaches an F1 of 0.68 on this benchmark, beating baselines including WEIR. It also finds far more relations than the seed ontology contains. This shows that templated sites can enrich knowledge bases with new relation types, not just new facts.

## Key points
- **Benchmark:** the expanded SWDE benchmark covers 21 websites in three domains, with 855,748 labels over 27,641 pages.
- **Ontology gap:** on 8 SWDE movie sites, the IMDb ontology covers only 7% of the semantically distinct predicates.
- **Predicate discovery:** OpenCeres reaches F1 0.68 and extracts roughly 7× as many predicates as the original ontology.
- **Candidate generation:** checking the k nearest higher-frequency fields yields recall of 0.92 (k=3, about 1,863 candidates) to 0.98 (k=11, about 4,104 candidates) on SWDE-Movie.
- **CommonCrawl results:** extraction worked on 22 of 31 movie sites. At confidence threshold 0.5, the system produced 2.3M triples at precision 0.58. At threshold 0.8, it produced 1.17M triples at precision 0.70, with about half using new predicates. The 9 failed sites were in foreign languages, had no predicate labels, or used terms outside the lexicon.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 175 works

## Cite

Cite as: Colin Lockard, Prashant Shiralkar and Xin Luna Dong. 2019. OpenCeres: When Open Information Extraction Meets the Semi-Structured Web. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lockard2019openceres,
  title = {{OpenCeres: When Open Information Extraction Meets the Semi-Structured Web}},
  author = {Colin Lockard, Prashant Shiralkar and Xin Luna Dong},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1309},
  url = {https://doi.org/10.18653/v1/n19-1309}
}
```

- [← previous Knowledge-aware Pronoun Coreference Resolution](https://md.research.elamur.ai/free/en/papers/zhang-2019-knowledge-aware-pronoun-coreference-resolution/)
- [next → A Neural Entity Coreference Resolution Review](https://md.research.elamur.ai/free/en/papers/stylianou-2019-neural-entity-coreference-resolution-review/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sreeram Balakrishnan, Alon Y. Halevy, Boulos Harb, Hongrae Lee, Jayant Madhavan, Afshin Rostamizadeh, Warren Shen, Kenneth Wilder, Fei Wu, and Cong Yu. 2015. Applying webtables in practice. In CIDR. [find](https://scholar.google.com/scholar?q=Sreeram+Balakrishnan%2C+Alon+Y%2E+Halevy%2C+Boulos+Harb%2C+Hongrae+Lee%2C+Jayant+Madhavan%2C+Afshin+Rostamizadeh%2C+Warren+Shen%2C+Kenneth+Wilder%2C+Fei+Wu%2C+and+Cong+Yu%2E+2015%2E+Applying+webtables+in+practice%2E+In+CIDR%2E)
2. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew G Broadhead, and Oren Etzioni. 2007. [Open information extraction from the web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/). In IJCAI.
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). TACL, 5:135–146.
4. Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. 2013. Extraction and integration of partially overlapping web sources. PVLDB, 6:805– 816. [find](https://scholar.google.com/scholar?q=Mirko+Bronzi%2C+Valter+Crescenzi%2C+Paolo+Merialdo%2C+and+Paolo+Papotti%2E+2013%2E+Extraction+and+integration+of+partially+overlapping+web+sources%2E+PVLDB%2C+6%3A805%E2%80%93+816%2E)
5. Michael J. Cafarella, Alon Y. Halevy, Hongrae Lee, Jayant Madhavan, Cong Yu, Daisy Zhe Wang, and Eugene Wu. 2018. Ten years of webtables. PVLDB, 11:2140–2149. [find](https://scholar.google.com/scholar?q=Michael+J%2E+Cafarella%2C+Alon+Y%2E+Halevy%2C+Hongrae+Lee%2C+Jayant+Madhavan%2C+Cong+Yu%2C+Daisy+Zhe+Wang%2C+and+Eugene+Wu%2E+2018%2E+Ten+years+of+webtables%2E+PVLDB%2C+11%3A2140%E2%80%932149%2E)
6. Michael J. Cafarella, Alon Y. Halevy, Daisy Zhe Wang, Eugene Wu, and Yonghui Zhang. 2008. Webtables: exploring the power of tables on the web. PVLDB, 1:538–549. [find](https://scholar.google.com/scholar?q=Michael+J%2E+Cafarella%2C+Alon+Y%2E+Halevy%2C+Daisy+Zhe+Wang%2C+Eugene+Wu%2C+and+Yonghui+Zhang%2E+2008%2E+Webtables%3A+exploring+the+power+of+tables+on+the+web%2E+PVLDB%2C+1%3A538%E2%80%93549%2E)
7. Valter Crescenzi, Giansalvatore Mecca, Paolo Merialdo, et al. 2001. Roadrunner: Towards automatic data extraction from large web sites. In VLDB, volume 1, pages 109–118. [find](https://scholar.google.com/scholar?q=Valter+Crescenzi%2C+Giansalvatore+Mecca%2C+Paolo+Merialdo%2C+et+al%2E+2001%2E+Roadrunner%3A+Towards+automatic+data+extraction+from+large+web+sites%2E+In+VLDB%2C+volume+1%2C+pages+109%E2%80%93118%2E)
8. Anna Lisa Gentile, Ziqi Zhang, and Fabio Ciravegna. 2015. Early steps towards web scale information extraction with lodie. AI Magazine, 36:55–64. [find](https://scholar.google.com/scholar?q=Anna+Lisa+Gentile%2C+Ziqi+Zhang%2C+and+Fabio+Ciravegna%2E+2015%2E+Early+steps+towards+web+scale+information+extraction+with+lodie%2E+AI+Magazine%2C+36%3A55%E2%80%9364%2E)
9. Pankaj Gulhane, Amit Madaan, Rupesh R. Mehta, Jeyashankher Ramamirtham, Rajeev Rastogi, Sandeepkumar Satpal, Srinivasan H. Sengamedu, Ashwin Tengli, and Charu Tiwari. 2011. Web-scale information extraction with vertex. ICDE, pages 1209–1220. [find](https://scholar.google.com/scholar?q=Pankaj+Gulhane%2C+Amit+Madaan%2C+Rupesh+R%2E+Mehta%2C+Jeyashankher+Ramamirtham%2C+Rajeev+Rastogi%2C+Sandeepkumar+Satpal%2C+Srinivasan+H%2E+Sengamedu%2C+Ashwin+Tengli%2C+and+Charu+Tiwari%2E+2011%2E+Web%2Dscale+information+extra)
10. Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. 2011. From one tree to a forest: a unified solution for structured web data extraction. In SIGIR. [find](https://scholar.google.com/scholar?q=Qiang+Hao%2C+Rui+Cai%2C+Yanwei+Pang%2C+and+Lei+Zhang%2E+2011%2E+From+one+tree+to+a+forest%3A+a+unified+solution+for+structured+web+data+extraction%2E+In+SIGIR%2E)
11. Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matt+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+EMNLP%2E)
12. Nicholas Kushmerick, Daniel S. Weld, and Robert B. Doorenbos. 1997. Wrapper induction for information extraction. In IJCAI. [find](https://scholar.google.com/scholar?q=Nicholas+Kushmerick%2C+Daniel+S%2E+Weld%2C+and+Robert+B%2E+Doorenbos%2E+1997%2E+Wrapper+induction+for+information+extraction%2E+In+IJCAI%2E)
13. Frank Lin and William W. Cohen. 2010. Semi-supervised classification of network data using very few labels. 2010 International Conference on Advances in Social Networks Analysis and Mining, pages 192–199. [find](https://scholar.google.com/scholar?q=Frank+Lin+and+William+W%2E+Cohen%2E+2010%2E+Semi%2Dsupervised+classification+of+network+data+using+very+few+labels%2E+2010+International+Conference+on+Advances+in+Social+Networks+Analysis+and+Mining%2C+pages+192%E2%80%93)
14. Colin Lockard, Xin Dong, Prashant Shiralkar, and Arash Einolghozati. 2018. Ceres: Distantly supervised relation extraction from the semi-structured web. PVLDB, 11:1084–1096. [find](https://scholar.google.com/scholar?q=Colin+Lockard%2C+Xin+Dong%2C+Prashant+Shiralkar%2C+and+Arash+Einolghozati%2E+2018%2E+Ceres%3A+Distantly+supervised+relation+extraction+from+the+semi%2Dstructured+web%2E+PVLDB%2C+11%3A1084%E2%80%931096%2E)
15. Mausam. 2016. Open information extraction systems and downstream applications. In IJCAI. [find](https://scholar.google.com/scholar?q=Mausam%2E+2016%2E+Open+information+extraction+systems+and+downstream+applications%2E+In+IJCAI%2E)
16. Lei Cui, Furu Wei, and Ming Zhou. 2018. [Neural open information extraction](https://md.research.elamur.ai/free/en/papers/cui-2018-neural-open-information-extraction/). In ACL.
17. Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learning for information extraction. In EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=Mausam%2C+Michael+Schmitz%2C+Stephen+Soderland%2C+Robert+Bart%2C+and+Oren+Etzioni%2E+2012%2E+Open+language+learning+for+information+extraction%2E+In+EMNLP%2DCoNLL%2E)
18. Bhavana Bharat Dalvi, William W. Cohen, and James P. Callan. 2012. Websets: extracting sets of entities from the web using unsupervised information extraction. In WSDM. [find](https://scholar.google.com/scholar?q=Bhavana+Bharat+Dalvi%2C+William+W%2E+Cohen%2C+and+James+P%2E+Callan%2E+2012%2E+Websets%3A+extracting+sets+of+entities+from+the+web+using+unsupervised+information+extraction%2E+In+WSDM%2E)
19. Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2018. A survey on open information extraction. In COLING. [find](https://scholar.google.com/scholar?q=Christina+Niklaus%2C+Matthias+Cetto%2C+Andre%CC%81+Freitas%2C+and+Siegfried+Handschuh%2E+2018%2E+A+survey+on+open+information+extraction%2E+In+COLING%2E)
20. Nilesh N. Dalvi, Ravi Kumar, and Mohamed A. Soliman. 2011. Automatic wrappers for large scale web extraction. PVLDB, 4:219–230. [find](https://scholar.google.com/scholar?q=Nilesh+N%2E+Dalvi%2C+Ravi+Kumar%2C+and+Mohamed+A%2E+Soliman%2E+2011%2E+Automatic+wrappers+for+large+scale+web+extraction%2E+PVLDB%2C+4%3A219%E2%80%93230%2E)
21. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In ACL.
22. Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In KDD. [find](https://scholar.google.com/scholar?q=Xin+Dong%2C+Evgeniy+Gabrilovich%2C+Geremy+Heitz%2C+Wilko+Horn%2C+Ni+Lao%2C+Kevin+Murphy%2C+Thomas+Strohmann%2C+Shaohua+Sun%2C+and+Wei+Zhang%2E+2014%2E+Knowledge+vault%3A+a+web%2Dscale+approach+to+probabilistic+knowledge+fusi)
23. Stephen Soderland. 1999. Learning information extraction rules for semi-structured and free text. Machine Learning, 34:233–272. [find](https://scholar.google.com/scholar?q=Stephen+Soderland%2E+1999%2E+Learning+information+extraction+rules+for+semi%2Dstructured+and+free+text%2E+Machine+Learning%2C+34%3A233%E2%80%93272%2E)
24. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In EMNLP. [find](https://scholar.google.com/scholar?q=Anthony+Fader%2C+Stephen+Soderland%2C+and+Oren+Etzioni%2E+2011%2E+Identifying+relations+for+open+information+extraction%2E+In+EMNLP%2E)
25. Gabriel Stanovsky, Julian Michael, Luke S. Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Julian+Michael%2C+Luke+S%2E+Zettlemoyer%2C+and+Ido+Dagan%2E+2018%2E+Supervised+open+information+extraction%2E+In+NAACL%2DHLT%2E)
26. Fei Wu and Daniel S. Weld. 2010. [Open information extraction using wikipedia](https://md.research.elamur.ai/free/en/papers/wu-2010-open-information-extraction-wikipedia/). In ACL.
27. Freebase. 2018. Compound value types in Freebase API. [find](https://scholar.google.com/scholar?q=Freebase%2E+2018%2E+Compound+value+types+in+Freebase+API%2E)
28. Tim Furche, Georg Gottlob, Giovanni Grasso, Xiaonan Guo, Giorgio Orsi, Christian Schallhart, and Cheng Wang. 2014. Diadem: Thousands of websites to a single database. PVLDB, 7:1845–1856. [find](https://scholar.google.com/scholar?q=Tim+Furche%2C+Georg+Gottlob%2C+Giovanni+Grasso%2C+Xiaonan+Guo%2C+Giorgio+Orsi%2C+Christian+Schallhart%2C+and+Cheng+Wang%2E+2014%2E+Diadem%3A+Thousands+of+websites+to+a+single+database%2E+PVLDB%2C+7%3A1845%E2%80%931856%2E)

*Updated 31 October 2026*