---
title: "Trafilatura — paper by Barbaresi (2021) — summary and ideas"
description: "The paper presents Trafilatura, an open-source Python library and command-line tool. 0.934 recall. 54 references, cited by 73 papers here."
authors: ["Adrien Barbaresi"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations"
source: "https://aclanthology.org/2021.acl-demo.15/"
doi: "10.18653/v1/2021.acl-demo.15"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/barbaresi-2021-trafilatura-web-scraping-library-command/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/barbaresi-2021-trafilatura-web-scraping-library-command/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

# Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

**Authors:** Adrien Barbaresi · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-demo.15/) · [PDF](https://aclanthology.org/2021.acl-demo.15.pdf) · [DOI 10.18653/v1/2021.acl-demo.15](https://doi.org/10.18653/v1/2021.acl-demo.15) · [OpenAlex](https://openalex.org/W3177057043)

**Citations:** 80 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/barbaresi-2021-trafilatura-web-scraping-library-command/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Building web corpora requires separating a page's useful text from surrounding noise such as navigation, ads, headers and footers, and finding relevant pages on a website. Existing open-source extractors did not meet the author's needs for accuracy, flexibility and ease of use. This was especially true for building text databases for a lexicographic platform at the ZDL/BBAW. The paper presents Trafilatura, an open-source Python library and command-line tool. It extracts main text, comments and metadata, and also provides components for text discovery and crawling, such as navigating sitemaps and web feeds. The author compares it with a range of other extraction packages on 500 real-world documents annotated at segment level, and also reports a simple, fast baseline. Trafilatura achieves the best balance of precision and recall and clearly outperforms the alternatives. Independent benchmarks confirm this result. For NLP and corpus linguistics, the tool offers a practical way to obtain cleaner web text, which helps with de-duplication, language modelling and close reading.

## Key points
- The benchmark covers 500 documents, with 1487 text and 1496 boilerplate segments.
- Trafilatura 0.8.2 reached an F-score of 0.912 (precision 0.934, recall 0.890), and its "fast" mode reached 0.896.
- The next-best tools, news-please (0.808) and readability-lxml (0.804), scored about 0.10 lower in F-score.
- In the external evaluations:
  - an earlier version topped ScrapingHub's article extraction benchmark among open-source libraries;
  - it also gave the best macro-mean on the multilingual DANIEL corpus (about 1,600 pages in five languages);
  - it performed well for French and Swedish and for building Russian-Turkic parallel corpora.
- The tool favours text quality over always returning output, so some pages yield no text. Planned work includes precision- and recall-oriented presets, evaluation of comment extraction, and a spider for websites that lack sitemaps or feeds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Berlin-Brandenburg Academy of Sciences and Humanities](https://ror.org/05jgq9443).

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works

## Cite

Cite as: Adrien Barbaresi. 2021. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{barbaresi2021trafilatura,
  title = {{Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction}},
  author = {Adrien Barbaresi},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.15},
  url = {https://doi.org/10.18653/v1/2021.acl-demo.15}
}
```

- [← previous Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://md.research.elamur.ai/free/en/papers/ross-2021-explaining-nlp-models-minimal-contrastive/)
- [next → Extractive Opinion Summarization in Quantized Transformer Spaces](https://md.research.elamur.ai/free/en/papers/angelidis-2021-extractive-opinion-summarization-quantized-transformer/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

54 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julian Alarte, Josep Silva, and Salvador Tamarit. 2019. What Web Template Extractor Should I Use?A Benchmarking and Comparison for Five Template Extractors. ACM Transactions on the Web (TWEB), 13(2):1–19. [find](https://scholar.google.com/scholar?q=Julian+Alarte%2C+Josep+Silva%2C+and+Salvador+Tamarit%2E+2019%2E+What+Web+Template+Extractor+Should+I+Use%3FA+Benchmarking+and+Comparison+for+Five+Template+Extractors%2E+ACM+Transactions+on+the+Web+%28TWEB%29%2C+13%282%29%3A1)
2. Adrien Barbaresi. 2015. Ad hoc and general-purpose corpus construction from web sources. Ph.D. thesis, École Normale Supérieure de Lyon. [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi%2E+2015%2E+Ad+hoc+and+general%2Dpurpose+corpus+construction+from+web+sources%2E+Ph%2ED%2E+thesis%2C+E%CC%81cole+Normale+Supe%CC%81rieure+de+Lyon%2E)
3. Adrien Barbaresi. 2016. Efficient construction of metadata-enhanced web corpora. In Proceedings of the 10th Web as Corpus Workshop, pages 7–16. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi%2E+2016%2E+Efficient+construction+of+metadata%2Denhanced+web+corpora%2E+In+Proceedings+of+the+10th+Web+as+Corpus+Workshop%2C+pages+7%E2%80%9316%2E+Association+for+Computational+Linguistics%2E)
4. Marco Brunello. 2009. The creation of free linguistic corpora from the web. In Proceedings of the Fifth Web as Corpus Workshop (WAC5), pages 9–16. Elhuyar Fundazioa. [find](https://scholar.google.com/scholar?q=Marco+Brunello%2E+2009%2E+The+creation+of+free+linguistic+corpora+from+the+web%2E+In+Proceedings+of+the+Fifth+Web+as+Corpus+Workshop+%28WAC5%29%2C+pages+9%E2%80%9316%2E+Elhuyar+Fundazioa%2E)
5. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. [N-gram Counts and Language Models from the Common Crawl](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/). In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC’14). ELRA.
6. Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2003. VIPS: a Vision-based Page Segmentation Algorithm. Technical report, Microsoft Technical Report (MSR-TR-2003-79). [find](https://scholar.google.com/scholar?q=Deng+Cai%2C+Shipeng+Yu%2C+Ji%2DRong+Wen%2C+and+Wei%2DYing+Ma%2E+2003%2E+VIPS%3A+a+Vision%2Dbased+Page+Segmentation+Algorithm%2E+Technical+report%2C+Microsoft+Technical+Report+%28MSR%2DTR%2D2003%2D79%29%2E)
7. Howard J. Carey and Milos Manic. 2016. HTML web content extraction using paragraph tags. In 25th International Symposium on Industrial Electronics (ISIE), pages 1099–1105. IEEE. [find](https://scholar.google.com/scholar?q=Howard+J%2E+Carey+and+Milos+Manic%2E+2016%2E+HTML+web+content+extraction+using+paragraph+tags%2E+In+25th+International+Symposium+on+Industrial+Electronics+%28ISIE%29%2C+pages+1099%E2%80%931105%2E+IEEE%2E)
8. Adrien Barbaresi. 2020. htmldate: A Python package to extract publication dates from web pages. Journal of Open Source Software, 5(51):2439. [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi%2E+2020%2E+htmldate%3A+A+Python+package+to+extract+publication+dates+from+web+pages%2E+Journal+of+Open+Source+Software%2C+5%2851%29%3A2439%2E)
9. Adrien Barbaresi and Gaël Lejeune. 2020. Out-of-the-Box and Into the Ditch? Multilingual Evaluation of Generic Text Extraction Tools. In Proceedings of the 12th Web as Corpus Workshop, pages 5–13. [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi+and+Gae%CC%88l+Lejeune%2E+2020%2E+Out%2Dof%2Dthe%2DBox+and+Into+the+Ditch%3F+Multilingual+Evaluation+of+Generic+Text+Extraction+Tools%2E+In+Proceedings+of+the+12th+Web+as+Corpus+Workshop%2C+pages+5%E2%80%9313%2E)
10. Adrien Barbaresi and Kay-Michael Würzner. 2014. For a fistful of blogs: Discovery and comparative benchmarking of republishable German content. In Proceedings of KONVENS 2014, NLP4CMC workshop, pages 2–10. Hildesheim University Press. [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi+and+Kay%2DMichael+Wu%CC%88rzner%2E+2014%2E+For+a+fistful+of+blogs%3A+Discovery+and+comparative+benchmarking+of+republishable+German+content%2E+In+Proceedings+of+KONVENS+2014%2C+NLP4CMC+workshop%2C+pages)
11. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky Wide Web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Silvia+Bernardini%2C+Adriano+Ferraresi%2C+and+Eros+Zanchetta%2E+2009%2E+The+WaCky+Wide+Web%3A+a+collection+of+very+large+linguistically+processed+web%2Dcrawled+corpora%2E+Language+Resources+and+Evalua)
12. Marco Baroni, Francis Chantree, Adam Kilgarriff, and Serge Sharoff. 2008. Cleaneval: a Competition for Cleaning Web Pages. In Proceedings of the 6th Conference on Language Resources and Evaluation (LREC’08), pages 638–643. ELRA. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2C+Francis+Chantree%2C+Adam+Kilgarriff%2C+and+Serge+Sharoff%2E+2008%2E+Cleaneval%3A+a+Competition+for+Cleaning+Web+Pages%2E+In+Proceedings+of+the+6th+Conference+on+Language+Resources+and+Evaluation+%28LR)
13. Daniel Bauer, Judith Degen, Xiaoye Deng, Priska Herger, Jan Gasthaus, Eugenie Giesbrecht, Lina Jansen, Christin Kalina, Thorben Kräger, Robert Märtin, Martin Schmidt, Simon Scholler, Johannes Steger, Egon Stemle, and Stefan Evert. 2007. FI-ASCO: Filtering the internet by automatic subtree classifi… [find](https://scholar.google.com/scholar?q=Daniel+Bauer%2C+Judith+Degen%2C+Xiaoye+Deng%2C+Priska+Herger%2C+Jan+Gasthaus%2C+Eugenie+Giesbrecht%2C+Lina+Jansen%2C+Christin+Kalina%2C+Thorben+Kra%CC%88ger%2C+Robert+Ma%CC%88rtin%2C+Martin+Schmidt%2C+Simon+Scholler%2C+Johannes+Steger)
14. Vladimı́r Benko. 2016. wo Years of Aranea: Increasing Counts and Tuning the Pipeline. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16), pages 4245– 4248. ELRA. [find](https://scholar.google.com/scholar?q=Vladim%C4%B1%CC%81r+Benko%2E+2016%2E+wo+Years+of+Aranea%3A+Increasing+Counts+and+Tuning+the+Pipeline%2E+In+Proceedings+of+the+10th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+pages+4245%E2%80%93+42)
15. Chris Biemann, Gerhard Heyer, Uwe Quasthoff, and Matthias Richter. 2007. The Leipzig Corpora Collection: Monolingual Corpora of Standard Size. In Proceedings of the Corpus Linguistics Conference. [find](https://scholar.google.com/scholar?q=Chris+Biemann%2C+Gerhard+Heyer%2C+Uwe+Quasthoff%2C+and+Matthias+Richter%2E+2007%2E+The+Leipzig+Corpora+Collection%3A+Monolingual+Corpora+of+Standard+Size%2E+In+Proceedings+of+the+Corpus+Linguistics+Conference%2E)
16. Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. [Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus](https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608. International Committee on Compu…
17. Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. 2001. Roadrunner: Towards Automatic Data Extraction From Large Web Sites. In Proceedings of the 27th VLDB Conference, pages 109–118. [find](https://scholar.google.com/scholar?q=Valter+Crescenzi%2C+Giansalvatore+Mecca%2C+and+Paolo+Merialdo%2E+2001%2E+Roadrunner%3A+Towards+Automatic+Data+Extraction+From+Large+Web+Sites%2E+In+Proceedings+of+the+27th+VLDB+Conference%2C+pages+109%E2%80%93118%2E)
18. Orphée De Clercq and Maribel Montero Perez. 2010. Data Collection and IPR in Multilingual Parallel Corpora. Dutch Parallel Corpus. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC’10), pages 3383– 3388. ELRA. [find](https://scholar.google.com/scholar?q=Orphe%CC%81e+De+Clercq+and+Maribel+Montero+Perez%2E+2010%2E+Data+Collection+and+IPR+in+Multilingual+Parallel+Corpora%2E+Dutch+Parallel+Corpus%2E+In+Proceedings+of+the+7th+International+Conference+on+Language+Resou)
19. Aidan Finn, Nicholas Kushmerick, and Barry Smyth. 2001. Fact or Fiction: Content Classification for Digital Libraries. In Joint DELOS-NSF Workshop: Personalization andRecommender Systems in Digital Libraries. [find](https://scholar.google.com/scholar?q=Aidan+Finn%2C+Nicholas+Kushmerick%2C+and+Barry+Smyth%2E+2001%2E+Fact+or+Fiction%3A+Content+Classification+for+Digital+Libraries%2E+In+Joint+DELOS%2DNSF+Workshop%3A+Personalization+andRecommender+Systems+in+Digital+Li)
20. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+Pile%3A+An+800GB+Dataset+of+Diverse+Text+for+Lan)
21. Alexander Geyken, Adrien Barbaresi, Jrg Didakowski, Bryan Jurish, Frank Wiegand, and Lothar Lemnitzer. 2017. Die Korpusplattform des ”Digitalen Wrter-buchs der deutschen Sprache” (DWDS). Zeitschrift fr germanistische Linguistik, 45(2):327–344. [find](https://scholar.google.com/scholar?q=Alexander+Geyken%2C+Adrien+Barbaresi%2C+Jrg+Didakowski%2C+Bryan+Jurish%2C+Frank+Wiegand%2C+and+Lothar+Lemnitzer%2E+2017%2E+Die+Korpusplattform+des+%E2%80%9DDigitalen+Wrter%2Dbuchs+der+deutschen+Sprache%E2%80%9D+%28DWDS%29%2E+Zeitschrift+f)
22. Mohammad Ghasemisharif, Peter Snyder, Andrius Aucinas, and Benjamin Livshits. 2019. SpeedReader: Reader Mode Made Fast and Private. In Proceedings of the World Wide Web Conference, pages 526–537. [find](https://scholar.google.com/scholar?q=Mohammad+Ghasemisharif%2C+Peter+Snyder%2C+Andrius+Aucinas%2C+and+Benjamin+Livshits%2E+2019%2E+SpeedReader%3A+Reader+Mode+Made+Fast+and+Private%2E+In+Proceedings+of+the+World+Wide+Web+Conference%2C+pages+526%E2%80%93537%2E)
23. Natalie Glance, Matthew Hurst, and Takashi Tomokiyo. 2004. Blogpulse: Automated Trend Discovery for Weblogs. In WWW 2004 Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics. [find](https://scholar.google.com/scholar?q=Natalie+Glance%2C+Matthew+Hurst%2C+and+Takashi+Tomokiyo%2E+2004%2E+Blogpulse%3A+Automated+Trend+Discovery+for+Weblogs%2E+In+WWW+2004+Workshop+on+the+Weblogging+Ecosystem%3A+Aggregation%2C+Analysis+and+Dynamics%2E)
24. Yan Guo, Huifeng Tang, Linhai Song, Yu Wang, and Guodong Ding. 2010. ECON: an Approach to Extract Content from Web News Page. In Proceedings of 12th International Asia-Pacific Web Conference (APWEB), pages 314–320. IEEE. [find](https://scholar.google.com/scholar?q=Yan+Guo%2C+Huifeng+Tang%2C+Linhai+Song%2C+Yu+Wang%2C+and+Guodong+Ding%2E+2010%2E+ECON%3A+an+Approach+to+Extract+Content+from+Web+News+Page%2E+In+Proceedings+of+12th+International+Asia%2DPacific+Web+Conference+%28APWEB%29%2C+)
25. Ivan Habernal, Omnia Zayed, and Iryna Gurevych. 2016. C4Corpus: Multilingual Web-size Corpus with Free License. In Proceedings of the 10th Conference on Language Resources and Evaluation (LREC’16), pages 914–922. [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Omnia+Zayed%2C+and+Iryna+Gurevych%2E+2016%2E+C4Corpus%3A+Multilingual+Web%2Dsize+Corpus+with+Free+License%2E+In+Proceedings+of+the+10th+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+pa)
26. Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223. [find](https://scholar.google.com/scholar?q=Felix+Hamborg%2C+Norman+Meuschke%2C+Corinna+Breitinger%2C+and+Bela+Gipp%2E+2017%2E+news%2Dplease%3A+A+Generic+News+Crawler+and+Extractor%2E+In+Proceedings+of+the+15th+International+Symposium+of+Information+Science%2C+p)
27. Hung-Yu Kao, Shian-Hua Lin, Jan-Ming Ho, and Ming-Syan Chen. 2004. Mining web informative structures and contents based on entropy analysis. IEEE Transactions on Knowledge and Data Engineering, 16(1):41–55. [find](https://scholar.google.com/scholar?q=Hung%2DYu+Kao%2C+Shian%2DHua+Lin%2C+Jan%2DMing+Ho%2C+and+Ming%2DSyan+Chen%2E+2004%2E+Mining+web+informative+structures+and+contents+based+on+entropy+analysis%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+16%281%29%3A)
28. Aidar Khusainov, Dzhavdet Suleymanov, Rinat Gilmullin, Alina Minsafina, Lenara Kubedinova, and Nilufar Abdurakhmonova. 2020. First Results of the TurkLang-7 Project: Creating Russian-Turkic Parallel Corpora and MT Systems. In Proceedings of the Computational Models in Language and Speech Workshop (C… [find](https://scholar.google.com/scholar?q=Aidar+Khusainov%2C+Dzhavdet+Suleymanov%2C+Rinat+Gilmullin%2C+Alina+Minsafina%2C+Lenara+Kubedinova%2C+and+Nilufar+Abdurakhmonova%2E+2020%2E+First+Results+of+the+TurkLang%2D7+Project%3A+Creating+Russian%2DTurkic+Parallel+C)
29. Adam Kilgarriff. 2007. Googleology is bad science. Computational Linguistics, 33(1):147–151. [find](https://scholar.google.com/scholar?q=Adam+Kilgarriff%2E+2007%2E+Googleology+is+bad+science%2E+Computational+Linguistics%2C+33%281%29%3A147%E2%80%93151%2E)
30. Christian Kohlschütter, Peter Fankhauser, and Wolfgang Nejdl. 2010. Boilerplate detection using shallow text features. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM 10, pages 441–450. [find](https://scholar.google.com/scholar?q=Christian+Kohlschu%CC%88tter%2C+Peter+Fankhauser%2C+and+Wolfgang+Nejdl%2E+2010%2E+Boilerplate+detection+using+shallow+text+features%2E+In+Proceedings+of+the+Third+ACM+International+Conference+on+Web+Search+and+Data+)
31. Julia Krasselt, Philipp Dressen, Matthias Fluor, Cerstin Mahlow, Klaus Rothenhäusler, and Maren Runte. 2020. Swiss-AL: A multilingual Swiss Web corpus for applied linguistics. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC’20), pages 4145–4151. ELRA. [find](https://scholar.google.com/scholar?q=Julia+Krasselt%2C+Philipp+Dressen%2C+Matthias+Fluor%2C+Cerstin+Mahlow%2C+Klaus+Rothenha%CC%88usler%2C+and+Maren+Runte%2E+2020%2E+Swiss%2DAL%3A+A+multilingual+Swiss+Web+corpus+for+applied+linguistics%2E+In+Proceedings+of+the+1)
32. JEP, TALN, RÉCITAL. Volume 4: Démonstrations et résumés d’articles internationaux, pages 46–49. Gaël Lejeune and Lichao Zhu. 2018. A New Proposal for Evaluating Web Page Cleaning Tools. Computación y Sistemas, 22(4). [find](https://scholar.google.com/scholar?q=JEP%2C+TALN%2C+RE%CC%81CITAL%2E+Volume+4%3A+De%CC%81monstrations+et+re%CC%81sume%CC%81s+d%E2%80%99articles+internationaux%2C+pages+46%E2%80%9349%2E+Gae%CC%88l+Lejeune+and+Lichao+Zhu%2E+2018%2E+A+New+Proposal+for+Evaluating+Web+Page+Cleaning+Tools%2E+Computaci)
33. Verena Lyding, Egon Stemle, Claudia Borghetti, Marco Brunello, Sara Castagnoli, Felice Dell’Orletta, Henrik Dittmann, Alessandro Lenci, and Vito Pirrelli. 2014. The PAISÁ Corpus of Italian Web Texts. In 9th Web as Corpus Workshop (WaC-9) @ EACL 2014, pages 36–43. European chapter of the Association… [find](https://scholar.google.com/scholar?q=Verena+Lyding%2C+Egon+Stemle%2C+Claudia+Borghetti%2C+Marco+Brunello%2C+Sara+Castagnoli%2C+Felice+Dell%E2%80%99Orletta%2C+Henrik+Dittmann%2C+Alessandro+Lenci%2C+and+Vito+Pirrelli%2E+2014%2E+The+PAISA%CC%81+Corpus+of+Italian+Web+Texts%2E)
34. Gilad Mishne and Natalie Glance. 2006. Leave a Reply: An Analysis of Weblog Comments. In Third Annual Workshop on the Weblogging Ecosystem, WWW 2006. [find](https://scholar.google.com/scholar?q=Gilad+Mishne+and+Natalie+Glance%2E+2006%2E+Leave+a+Reply%3A+An+Analysis+of+Weblog+Comments%2E+In+Third+Annual+Workshop+on+the+Weblogging+Ecosystem%2C+WWW+2006%2E)
35. Tomoyuki Nanno, Toshiaki Fujiki, Yasuhiro Suzuki, and Manabu Okumura. 2004. Automatically Collecting, Monitoring, and Mining Japanese Weblogs. In Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, pages 320–321. ACM. [find](https://scholar.google.com/scholar?q=Tomoyuki+Nanno%2C+Toshiaki+Fujiki%2C+Yasuhiro+Suzuki%2C+and+Manabu+Okumura%2E+2004%2E+Automatically+Collecting%2C+Monitoring%2C+and+Mining+Japanese+Weblogs%2E+In+Proceedings+of+the+13th+international+World+Wide+Web+c)
36. Christopher Olston and Marc Najork. 2010. Web Crawling. Foundations and Trends in Information Retrieval, 4(3):175–246. [find](https://scholar.google.com/scholar?q=Christopher+Olston+and+Marc+Najork%2E+2010%2E+Web+Crawling%2E+Foundations+and+Trends+in+Information+Retrieval%2C+4%283%29%3A175%E2%80%93246%2E)
37. Jeff Pasternack and Dan Roth. 2009. Extracting article text from the web with maximum subsequence segmentation. In Proceedings of the 18th international conference on World Wide Web, pages 971–980. [find](https://scholar.google.com/scholar?q=Jeff+Pasternack+and+Dan+Roth%2E+2009%2E+Extracting+article+text+from+the+web+with+maximum+subsequence+segmentation%2E+In+Proceedings+of+the+18th+international+conference+on+World+Wide+Web%2C+pages+971%E2%80%93980%2E)
38. Matthew E. Peters and Dan Lecocq. 2013. Content extraction using diverse feature sets. In Proceedings of the 22nd International Conference on World Wide Web, pages 89–90. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters+and+Dan+Lecocq%2E+2013%2E+Content+extraction+using+diverse+feature+sets%2E+In+Proceedings+of+the+22nd+International+Conference+on+World+Wide+Web%2C+pages+89%E2%80%9390%2E)
39. Jan Pomikálek. 2011. Removing boilerplate and duplicate content from web corpora. Ph.D. thesis, Masaryk University. [find](https://scholar.google.com/scholar?q=Jan+Pomika%CC%81lek%2E+2011%2E+Removing+boilerplate+and+duplicate+content+from+web+corpora%2E+Ph%2ED%2E+thesis%2C+Masaryk+University%2E)
40. Pir Abdul Rasool Qureshi and Nasrullah Memon. 2012. Hybrid model of content extraction. Journal of Computer and System Sciences, 78(4):1248–1257. [find](https://scholar.google.com/scholar?q=Pir+Abdul+Rasool+Qureshi+and+Nasrullah+Memon%2E+2012%2E+Hybrid+model+of+content+extraction%2E+Journal+of+Computer+and+System+Sciences%2C+78%284%29%3A1248%E2%80%931257%2E)
41. Kevin P. Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. In Proceedings of the 3rd Web as Corpus Workshop, volume 4, pages 5–15, Louvain-la-Neuve. [find](https://scholar.google.com/scholar?q=Kevin+P%2E+Scannell%2E+2007%2E+The+Cru%CC%81bada%CC%81n+Project%3A+Corpus+building+for+under%2Dresourced+languages%2E+In+Proceedings+of+the+3rd+Web+as+Corpus+Workshop%2C+volume+4%2C+pages+5%E2%80%9315%2C+Louvain%2Dla%2DNeuve%2E)
42. Veronika Laippala, Samuel Rönnqvist, Saara Hell-ström, Juhani Luotolahti, Liina Repo, Anna Salmela, Valtteri Skantsi, and Sampo Pyysalo. 2020. From Web Crawl to Clean Register-Annotated Corpora. In Proceedings of the 12th Web as Corpus Workshop, pages 14–22. [find](https://scholar.google.com/scholar?q=Veronika+Laippala%2C+Samuel+Ro%CC%88nnqvist%2C+Saara+Hell%2Dstro%CC%88m%2C+Juhani+Luotolahti%2C+Liina+Repo%2C+Anna+Salmela%2C+Valtteri+Skantsi%2C+and+Sampo+Pyysalo%2E+2020%2E+From+Web+Crawl+to+Clean+Register%2DAnnotated+Corpora%2E+In+)
43. Roland Schäfer. 2016. CommonCOW: massively huge web corpora from CommonCrawl data and a method to distribute them freely under restrictive EU copyright laws. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16), pages 4500–4504. ELRA. [find](https://scholar.google.com/scholar?q=Roland+Scha%CC%88fer%2E+2016%2E+CommonCOW%3A+massively+huge+web+corpora+from+CommonCrawl+data+and+a+method+to+distribute+them+freely+under+restrictive+EU+copyright+laws%2E+In+Proceedings+of+the+10th+International+)
44. Gaël Lejeune and Adrien Barbaresi. 2020. Bien choisir son outil d’extraction de contenu à partir du Web (Choosing the appropriate tool for Web Content Extraction). In Actes de la 6e conférence conjointe Roland Schäfer. 2017. Accurate and efficient general-purpose boilerplate detection for crawle… [find](https://scholar.google.com/scholar?q=Gae%CC%88l+Lejeune+and+Adrien+Barbaresi%2E+2020%2E+Bien+choisir+son+outil+d%E2%80%99extraction+de+contenu+a%CC%80+partir+du+Web+%28Choosing+the+appropriate+tool+for+Web+Content+Extraction%29%2E+In+Actes+de+la+6e+confe%CC%81rence+conj)
45. Roland Schäfer, Adrien Barbaresi, and Felix Bildhauer. 2013. The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction. In Proceedings of the 8th Web as Corpus Workshop, pages 7–15. [find](https://scholar.google.com/scholar?q=Roland+Scha%CC%88fer%2C+Adrien+Barbaresi%2C+and+Felix+Bildhauer%2E+2013%2E+The+Good%2C+the+Bad%2C+and+the+Hazy%3A+Design+Decisions+in+Web+Corpus+Construction%2E+In+Proceedings+of+the+8th+Web+as+Corpus+Workshop%2C+pages+7%E2%80%9315)
46. Roland Schäfer and Felix Bildhauer. 2012. [Building Large Corpora from the Web Using a New Efficient Tool Chain](https://md.research.elamur.ai/free/en/papers/schafer-2012-building-large-corpora-web-new/). In Proceedings of the 8th Conference on Language Resources and Evaluation (LREC’12), pages 486–493. ELRA.
47. Miroslav Spousta, Michal Marek, and Pavel Pecina. 2008. Victor: the Web-Page Cleaning Tool. In 4th Web as Corpus Workshop (WAC-4), pages 12–17. [find](https://scholar.google.com/scholar?q=Miroslav+Spousta%2C+Michal+Marek%2C+and+Pavel+Pecina%2E+2008%2E+Victor%3A+the+Web%2DPage+Cleaning+Tool%2E+In+4th+Web+as+Corpus+Workshop+%28WAC%2D4%29%2C+pages+12%E2%80%9317%2E)
48. Fei Sun, Dandan Song, and Lejian Liao. 2011. DOM-based content extraction via text density. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 245–254. [find](https://scholar.google.com/scholar?q=Fei+Sun%2C+Dandan+Song%2C+and+Lejian+Liao%2E+2011%2E+DOM%2Dbased+content+extraction+via+text+density%2E+In+Proceedings+of+the+34th+international+ACM+SIGIR+conference+on+Research+and+development+in+Information+Ret)
49. Karane Vieira, Altigran S Da Silva, Nick Pinto, Edleno S De Moura, Joao MB Cavalcanti, and Juliana Freire. 2006. A Fast and Robust Method for Web Page TemplateDetection and Removal. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pages 258–267. [find](https://scholar.google.com/scholar?q=Karane+Vieira%2C+Altigran+S+Da+Silva%2C+Nick+Pinto%2C+Edleno+S+De+Moura%2C+Joao+MB+Cavalcanti%2C+and+Juliana+Freire%2E+2006%2E+A+Fast+and+Robust+Method+for+Web+Page+TemplateDetection+and+Removal%2E+In+Proceedings+of+)
50. Thijs Vogels, Octavian-Eugen Ganea, and Carsten Eickhoff. 2018. Web2text: Deep structured boilerplate removal. In European Conference on Information Retrieval, pages 167–179. Springer. [find](https://scholar.google.com/scholar?q=Thijs+Vogels%2C+Octavian%2DEugen+Ganea%2C+and+Carsten+Eickhoff%2E+2018%2E+Web2text%3A+Deep+structured+boilerplate+removal%2E+In+European+Conference+on+Information+Retrieval%2C+pages+167%E2%80%93179%2E+Springer%2E)
51. Junfeng Wang, Xiaofei He, Can Wang, Jian Pei, Jiajun Bu, Chun Chen, Ziyu Guan, and Gang Lu. 2009. News Article Extraction with Template-Independent Wrapper. In Proceedings of the WWW 2009, pages 1085–1086. ACM. [find](https://scholar.google.com/scholar?q=Junfeng+Wang%2C+Xiaofei+He%2C+Can+Wang%2C+Jian+Pei%2C+Jiajun+Bu%2C+Chun+Chen%2C+Ziyu+Guan%2C+and+Gang+Lu%2E+2009%2E+News+Article+Extraction+with+Template%2DIndependent+Wrapper%2E+In+Proceedings+of+the+WWW+2009%2C+pages+1085%E2%80%93)
52. Tim Weninger, William H Hsu, and Jiawei Han. 2010. CETR: content extraction via tag ratios. In Proceedings of the 19th international conference on World Wide Web, pages 971–980. [find](https://scholar.google.com/scholar?q=Tim+Weninger%2C+William+H+Hsu%2C+and+Jiawei+Han%2E+2010%2E+CETR%3A+content+extraction+via+tag+ratios%2E+In+Proceedings+of+the+19th+international+conference+on+World+Wide+Web%2C+pages+971%E2%80%93980%2E)
53. Lan Yi, Bing Liu, and Xiaoli Li. 2003. Eliminating noisy information in web pages for data mining. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 296–305. [find](https://scholar.google.com/scholar?q=Lan+Yi%2C+Bing+Liu%2C+and+Xiaoli+Li%2E+2003%2E+Eliminating+noisy+information+in+web+pages+for+data+mining%2E+In+Proceedings+of+the+9th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining%2C)
54. Cai-Nicolas Ziegler and Michal Skubacz. 2007. Content Extraction from News Pages using Particle Swarm Optimization on Linguistic and Structural Features. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, pages 242–249. IEEE. [find](https://scholar.google.com/scholar?q=Cai%2DNicolas+Ziegler+and+Michal+Skubacz%2E+2007%2E+Content+Extraction+from+News+Pages+using+Particle+Swarm+Optimization+on+Linguistic+and+Structural+Features%2E+In+Proceedings+of+the+IEEE%2FWIC%2FACM+Internation)

*Updated 24 October 2026*