--- title: "Large-Scale Multi-Label Text Classification — Chalkidis 2019" description: "The paper addresses large-scale multi-label text classification (LMTC) for legal documents, where each text must receive every relevant tag from a label…" authors: ["Ilias Chalkidis", "Emmanouil Fergadiotis", "Prodromos Malakasiotis", "Ion Androutsopoulos"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1636/" doi: "10.18653/v1/p19-1636" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chalkidis-2019-large-scale-multi-label-text/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chalkidis-2019-large-scale-multi-label-text/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Large-Scale Multi-Label Text Classification on EU Legislation # Large-Scale Multi-Label Text Classification on EU Legislation **Authors:** [Ilias Chalkidis](https://md.research.elamur.ai/free/en/authors/ilias-chalkidis/), Emmanouil Fergadiotis, [Prodromos Malakasiotis](https://md.research.elamur.ai/free/en/authors/prodromos-malakasiotis/), [Ion Androutsopoulos](https://md.research.elamur.ai/free/en/authors/ion-androutsopoulos/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1636/) · [PDF](https://aclanthology.org/P19-1636.pdf) · [DOI 10.18653/v1/p19-1636](https://doi.org/10.18653/v1/p19-1636) · [OpenAlex](https://openalex.org/W2948729947) **Citations:** 191 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses large-scale multi-label text classification (LMTC) for legal documents, where each text must receive every relevant tag from a label set numbering in the thousands. The authors release EURLEX57K, a public corpus of 57k English EU legislative acts from EUR-LEX. Each act carries several EUROVOC concepts, and many of these concepts are rare or never appear in training. This makes the corpus suitable for testing frequent, few-shot and zero-shot labels separately. They compare simple baselines, attention-based RNNs, HAN, label-wise attention networks (CNN-LWAN and their own BIGRU-LWAN), descriptor-aware zero-shot variants and a fine-tuned BERT. A recurrent encoder with one attention head per label outperforms the earlier CNN-based design. Domain-trained WORD2VEC and ELMO embeddings add further gains. Because the title and recitals carry nearly all the signal, BERT can be fine-tuned on the first 512 tokens, and it gives the best scores everywhere except on zero-shot labels. The work supplies a modern legal benchmark, strong baselines and a fairer evaluation metric. ## Key points - **EURLEX57K dataset** - Contains 45k training, 6k development and 6k test documents. - Documents average about 727 words and carry about 5 labels each. - 4,271 of the roughly 7k EUROVOC concepts occur in the corpus. - Labels are grouped into 746 frequent (more than 50 training documents), 3,362 few-shot and 163 zero-shot. - **Full-document results** - BIGRU-LWAN performs best on full documents (RP@5 0.766 on all labels with GLOVE). - The generic BIGRU-ATT beats CNN-LWAN, which suggests that the CNN encoder is the weak part of that model. - **Embeddings** - Legal-domain WORD2VEC (L2V) combined with ELMO raises BIGRU-LWAN to RP@5 0.781 and Micro-F1 0.719. - **BERT-BASE (first 512 tokens)** - Scores RP@5 0.796, nDCG@5 0.823 and Micro-F1 0.732 on all labels. - According to the authors, no one had previously used BERT for large-scale multi-label classification. - **Zero-shot labels** - Descriptor-based models win here; ZERO-BIGRU-LWAN reaches RP@5 0.438, while BERT gets only 0.028. ## Interesting ideas - **RP@K metric:** this is precision at K, except that K is lowered to the number of gold labels whenever a document has fewer than K. It avoids penalising systems on documents with few tags, which helps any multi-label task where label counts vary per item. - **Zone ablation before truncation:** the authors first measured which document parts (header, recitals, main body) actually matter. This showed that cutting text to fit a 512-token model loses little. The same check could justify input truncation for long structured documents such as laws, contracts or reports. - **Frozen label-descriptor embeddings for unseen classes:** each label is represented by the average word vector of its descriptor, and these vectors are kept fixed. Because this prior knowledge is preserved, labels with no training examples can still be predicted, which is useful wherever taxonomies are large and long-tailed. - **Swapping one component against a strong baseline:** comparing the specialised CNN-LWAN with a plain attention BiGRU, and then replacing only its encoder, revealed exactly where the weakness lay. This kind of isolated swap is a cheap way to diagnose published architectures. - **Reporting results by label frequency:** scores are given separately for frequent, few-shot and zero-shot labels. This exposes trade-offs that a single overall score hides; for example, BERT excels on frequent labels but fails on unseen ones. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Law](https://md.research.elamur.ai/free/en/shelves/law/) ## Cited by Cited by 92 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 87 not yet. - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/) - [MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/) - [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-cuad-expert-annotated-nlp-dataset/) - [LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts](https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/) - [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://md.research.elamur.ai/free/en/papers/zhong-2020-how-does-nlp-benefit-legal/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{chalkidis2019large, title = {{Large-Scale Multi-Label Text Classification on EU Legislation}}, author = {Chalkidis, Ilias and Fergadiotis, Emmanouil and Malakasiotis, Prodromos and Androutsopoulos, Ion}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1636}, url = {https://doi.org/10.18653/v1/p19-1636} } ``` - [← previous Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) - [next → Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Nikolaos Aletras et al. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93. *Table 4: BIGRU - LWAN with different document zones.* [find](https://scholar.google.com/scholar?q=Nikolaos+Aletras+et+al%2E+2016%2E+Predicting+judicial+decisions+of+the+European+Court+of+Human+Rights%3A+a+Natural+Language+Processing+perspective%2E+PeerJ+Computer+Science%2C+2%3Ae93%2E+%2ATable+4%3A+BIGRU+%2D+LWAN+with)