---
title: "MultiEURLEX - A multi-lingual and — Chalkidis 2021"
description: "Cross-lingual transfer had not been studied in legal NLP."
authors: ["Ilias Chalkidis", "Manos Fergadiotis", "Ion Androutsopoulos"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.559/"
doi: "10.18653/v1/2021.emnlp-main.559"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

# MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

**Authors:** [Ilias Chalkidis](https://md.research.elamur.ai/free/en/authors/ilias-chalkidis/), [Manos Fergadiotis](https://md.research.elamur.ai/free/en/authors/manos-fergadiotis/), [Ion Androutsopoulos](https://md.research.elamur.ai/free/en/authors/ion-androutsopoulos/) · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.559/) · [PDF](https://aclanthology.org/2021.emnlp-main.559.pdf) · [DOI 10.18653/v1/2021.emnlp-main.559](https://doi.org/10.18653/v1/2021.emnlp-main.559) · [OpenAlex](https://openalex.org/W3198651167)

**Citations:** 84 citations (openalex-snapshot · 2026-10-14) · [76](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Cross-lingual transfer had not been studied in legal NLP. The authors address this with MultiEURLEX, a parallel corpus of about 65k EU laws in 23 official languages. Each law carries EUROVOC concepts assigned by the EU Publications Office, so the task is multi-label topic classification. Labels are offered at several levels of the hierarchy, and the data is split by date rather than at random.

The main use of the corpus is a zero-shot test: train on one language (English), then classify documents in the other 22. Fully fine-tuning XLM-ROBERTA or MT5 on a single language erases much of their multilingual knowledge, and transfer suffers. Lighter adaptation methods keep more of that knowledge and transfer better. These include:
- freezing some layers
- adapters
- BITFIT
- LNFIT

The benefit grows with the size of the label set. Even methods that train under 0.05% of the parameters stay competitive. The resource gives strong baselines for multilingual, multi-label classification, and it also shows how random splits can hide temporal concept drift.

## Key points
- **Coverage:** 65k EU laws in 23 languages from 7 families. All use Latin script except Bulgarian (Cyrillic) and Greek. Irish is excluded.
- **Released on Hugging Face** (`multi_eurlex`) with a Dataset Card, under CC BY 4.0. It extends the English-only EURLEX57K.
- **Label sets:** four granularities, with 21 (level 1), 127 (level 2), 567 (level 3) and 7,390 (original) concepts. Only about 57% of the original concepts occur in the training documents.
- **Chronological split:** training 55k (1958–2010), development 5k (2010–2012), test 5k (2012–2016). The same 5k test laws exist in every language. Training size varies by language, because later EU members have fewer documents.
- **Effect of label-set size (XLM-ROBERTA, trained on English):**
  - Averaged over adaptation strategies, the zero-shot gain over end-to-end fine-tuning rises from about +0.3 at level 1 to +4.2 at level 3, then dips slightly to +3.6 on the full label set.
  - The gap between English and the average over all languages widens from about −6.4 to −15.8 as labels get finer.
  - Adapters perform best overall.
  - Freezing all 12 blocks does not help.

## Interesting ideas
- **Reuse efficiency tricks to prevent forgetting.** Parameter-efficient methods (adapters, bias-only or LayerNorm-only tuning) were designed to make fine-tuning cheaper. Here they also protect what the model learned in pretraining. The same idea could apply to any transfer where the pretrained knowledge matters more than the source task, such as a high-resource-to-low-resource language pair like Ukrainian.
- **Derive coarser label sets from a taxonomy.** Replacing each gold concept with its ancestor at a chosen level turns one annotation into several tasks of increasing difficulty. This lets you see how a method's behaviour changes as labels become finer, without any new annotation.
- **Split by time, not at random.** For data like legislation or trade records, where topics change over time, a chronological split reveals concept drift. A random split would give an optimistic estimate.
- **Share one test set across languages.** Using the same parallel test documents for every language isolates language effects from content differences. Cross-lingual scores can then be compared fairly.
- **Sample a random language per batch.** For joint many-to-many training on uneven multilingual data, drawing an available language version of each document per batch is a simple scheme. It yields a single model that rivals separate monolingual models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Law](https://md.research.elamur.ai/free/en/shelves/law/)

## Cited by

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 70 not yet.

- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
- [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://md.research.elamur.ai/free/en/papers/niklaus-2023-lextreme-multi-lingual-multi-task/)
- [GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training](https://md.research.elamur.ai/free/en/papers/oort-2026-gpt-nl-public-corpus-permissively/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chalkidis2021multieurlex,
  title = {{MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer}},
  author = {Chalkidis, Ilias and Fergadiotis, Manos and Androutsopoulos, Ion},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.559},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.559}
}
```

- [← previous Multi-tasking Dialogue Comprehension with Discourse Parsing](https://md.research.elamur.ai/free/en/papers/he-2021-multi-tasking-dialogue-comprehension-discourse/)
- [next → Multilingual Coreference Resolution with Harmonized Annotations](https://md.research.elamur.ai/free/en/papers/prazak-2021-multilingual-coreference-resolution-harmonized-annotations/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

62 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nikolaos Aletras, Elliott Ash, Leslie Barrett, Daniel Chen, Adam Meyers, Daniel Preotiuc-Pietro, David Rosenberg, and Amanda Stent, editors. 2019. Proceedings of the Natural Legal Language Processing Workshop 2019. Minneapolis, Minnesota. [find](https://scholar.google.com/scholar?q=Nikolaos+Aletras%2C+Elliott+Ash%2C+Leslie+Barrett%2C+Daniel+Chen%2C+Adam+Meyers%2C+Daniel+Preotiuc%2DPietro%2C+David+Rosenberg%2C+and+Amanda+Stent%2C+editors%2E+2019%2E+Proceedings+of+the+Natural+Legal+Language+Processing+)
2. Nikolaos Aletras et al. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93. [find](https://scholar.google.com/scholar?q=Nikolaos+Aletras+et+al%2E+2016%2E+Predicting+judicial+decisions+of+the+European+Court+of+Human+Rights%3A+a+Natural+Language+Processing+perspective%2E+PeerJ+Computer+Science%2C+2%3Ae93%2E)
3. Iosif Angelidis, Ilias Chalkidis, and Manolis Koubarakis. 2018. Named Entity Recognition, Linking and Generation for Greek Legislation. In [https://innovationsfonden.dk/en](https://innovationsfonden.dk/en) Proceedings of the 31st International Conference on Legal Knowledge and Information Systems (JURIX), Groningen, The Netherlands. [↗](https://innovationsfonden.dk/en)
4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
5. Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+Multilingual+Sentence+Embeddings+for+Zero%2DShot+Cross%2DLingual+Transfer+and+Beyond%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A597) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
6. Jimmy Lei Ba, Jamie R. Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. In NIPS 2016 Deep Learning Symposium. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+R%2E+Kiros%2C+and+Geoffrey+E%2E+Hinton%2E+2016%2E+Layer+normalization%2E+In+NIPS+2016+Deep+Learning+Symposium%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
7. Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019. Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6314–6322, Florence, Italy. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Emmanouil+Fergadiotis%2C+Prodromos+Malakasiotis%2C+and+Ion+Androutsopoulos%2E+2019%2E+Large%2Dscale+multi%2Dlabel+text+classification+on+EU+legislation%2E+In+Proceedings+of+the+57th+Annual+Meeting+) [our card](https://md.research.elamur.ai/free/en/papers/chalkidis-2019-large-scale-multi-label-text/)
8. Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020a. An empirical study on large-scale multi-label text classification including few and zero-shot labels. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7503–7515, Online. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Manos+Fergadiotis%2C+Sotiris+Kotitsas%2C+Prodromos+Malakasiotis%2C+Nikolaos+Aletras%2C+and+Ion+Androutsopoulos%2E+2020a%2E+An+empirical+study+on+large%2Dscale+multi%2Dlabel+text+classification+includ)
9. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020b. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Manos+Fergadiotis%2C+Prodromos+Malakasiotis%2C+Nikolaos+Aletras%2C+and+Ion+Androutsopoulos%2E+2020b%2E+LEGAL%2DBERT%3A+The+muppets+straight+out+of+law+school%2E+In+Findings+of+the+Association+for+Com) [our card](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/)
10. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Online. [find](https://scholar.google.com/scholar?q=Branden+Chan%2C+Stefan+Schweter%2C+and+Timo+M%C3%B6ller%2E+2020%2E+German%E2%80%99s+next+language+model%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+6788%E2%80%936796%2C+Online%2E) [our card](https://md.research.elamur.ai/free/en/papers/chan-2020-german-s-next-language-model/)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
13. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255–3265, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pieter+Delobelle%2C+Thomas+Winters%2C+and+Bettina+Berendt%2E+2020%2E+RobBERT%3A+a+Dutch+RoBERTa%2Dbased+Language+Model%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+3255%E2%80%933265%2C+O) [our card](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
15. Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5702–5707, Hong Kong, China. [find](https://scholar.google.com/scholar?q=Julian+Eisenschlos%2C+Sebastian+Ruder%2C+Piotr+Czapla%2C+Marcin+Kadras%2C+Sylvain+Gugger%2C+and+Jeremy+Howard%2E+2019%2E+MultiFiT%3A+Efficient+multi%2Dlingual+language+model+fine%2Dtuning%2E+In+Proceedings+of+the+2019+Conf)
16. Task force of the European Commision. 2012. Special Eurobarometer 386: Europeans and their Languages. EU Directorate-General for Communication. [find](https://scholar.google.com/scholar?q=Task+force+of+the+European+Commision%2E+2012%2E+Special+Eurobarometer+386%3A+Europeans+and+their+Languages%2E+EU+Directorate%2DGeneral+for+Communication%2E)
17. Jonathan Frankle, David J. Schwab, and Ari S. Morcos. 2021. Training batchnorm and only batchnorm: On the expressive power of random features in cnns. In 9th International Conference on Learning Representations (ICLR 2021), Online. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle%2C+David+J%2E+Schwab%2C+and+Ari+S%2E+Morcos%2E+2021%2E+Training+batchnorm+and+only+batchnorm%3A+On+the+expressive+power+of+random+features+in+cnns%2E+In+9th+International+Conference+on+Learning+Repre)
18. Andrea Galassi, Kasper Drazewski, Marco Lippi, and Paolo Torroni. 2020. Cross-lingual annotation projection in legal texts. In Proceedings of the 28th International Conference on Computational Linguistics, pages 915–926, Barcelona, Spain (Online). [find](https://scholar.google.com/scholar?q=Andrea+Galassi%2C+Kasper+Drazewski%2C+Marco+Lippi%2C+and+Paolo+Torroni%2E+2020%2E+Cross%2Dlingual+annotation+projection+in+legal+texts%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguis)
19. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. CoRR, abs/1803.09010. [↗](https://arxiv.org/abs/1803.09010)
20. Teresa Gonalves and Paulo Quaresma. 2010. Multilingual text classification through combination of monolingual classifiers. In CEUR Workshop, volume 605. [find](https://scholar.google.com/scholar?q=Teresa+Gonalves+and+Paulo+Quaresma%2E+2010%2E+Multilingual+text+classification+through+combination+of+monolingual+classifiers%2E+In+CEUR+Workshop%2C+volume+605%2E)
21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770– 778, Las Vegas, NV, USA. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+2016+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+770%E2%80%93+778%2C+La)
22. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268. [↗](https://arxiv.org/abs/2103.06268) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-cuad-expert-annotated-nlp-dataset/)
23. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, CA, USA. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
24. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
25. Xiaolei Huang and Michael J. Paul. 2019. Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4113–4123, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaolei+Huang+and+Michael+J%2E+Paul%2E+2019%2E+Neural+temporality+adaptation+for+document+classification%3A+Diachronic+word+embeddings+and+domain+adaptation+models%2E+In+Proceedings+of+the+57th+Annual+Meeting+o)
26. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167. [↗](https://arxiv.org/abs/1502.03167)
27. Mi-young Kim, Ying Xu, and Randy Goebel. 2015. A Convolutional Neural Network in Legal Question Answering. Ninth International Workshop on Juris-informatics (JURISIN). [find](https://scholar.google.com/scholar?q=Mi%2Dyoung+Kim%2C+Ying+Xu%2C+and+Randy+Goebel%2E+2015%2E+A+Convolutional+Neural+Network+in+Legal+Question+Answering%2E+Ninth+International+Workshop+on+Juris%2Dinformatics+%28JURISIN%29%2E)
28. Diederik P. Kingma and Jim Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jim+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representations+%28ICLR%29%2C+San+Diego%2C+CA%2C+USA%2E)
29. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791, Florence, Italy. [find](https://scholar.google.com/scholar?q=Kyle+Gorman+and+Steven+Bedrick%2E+2019%2E+We+need+to+talk+about+standard+splits%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+2786%E2%80%932791%2C+Florence%2C+Ital)
30. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. 2021. WILDS: A benchmark of in-the-wild distribution shifts. CoRR, abs/2012.07421. [↗](https://arxiv.org/abs/2012.07421)
31. Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquín Silveira-Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller, Carlos Rodríguez Penagos, and Marta Villegas. 2021. Spanish language models. CoRR, abs/2107.07253. [↗](https://arxiv.org/abs/2107.07253)
32. John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. Greek-bert: The greeks visiting sesame street. In 11th Hellenic Conference on Artificial Intelligence, SETN 2020, page 110–117, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=John+Koutsikakis%2C+Ilias+Chalkidis%2C+Prodromos+Malakasiotis%2C+and+Ion+Androutsopoulos%2E+2020%2E+Greek%2Dbert%3A+The+greeks+visiting+sesame+street%2E+In+11th+Hellenic+Conference+on+Artificial+Intelligence%2C+SETN+20)
33. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Sebastian Ruder, Dani Yogatama, Kris Cao, Tomás Kociský, Susannah Young, and Phil Blunsom. 2021. Pitfalls of static language modelling. CoRR, abs/2102.01951. [↗](https://arxiv.org/abs/2102.01951)
34. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E)
35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
36. Pedro Henrique Luz de Araujo, Teófilo Emídio de Campos, Fabricio Ataides Braz, and Nilton Correia da Silva. 2020. VICTOR: a dataset for Brazilian legal documents classification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1449–1458, Marseille, France. [find](https://scholar.google.com/scholar?q=Pedro+Henrique+Luz+de+Araujo%2C+Te%C3%B3filo+Em%C3%ADdio+de+Campos%2C+Fabricio+Ataides+Braz%2C+and+Nilton+Correia+da+Silva%2E+2020%2E+VICTOR%3A+a+dataset+for+Brazilian+legal+documents+classification%2E+In+Proceedings+of+the+)
37. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2009. Introduction to Information Retrieval. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Prabhakar+Raghavan%2C+and+Hinrich+Sch%C3%BCtze%2E+2009%2E+Introduction+to+Information+Retrieval%2E+Cambridge+University+Press%2E)
38. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. [find](https://scholar.google.com/scholar?q=Louis+Martin%2C+Benjamin+Muller%2C+Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Yoann+Dupont%2C+Laurent+Romary%2C+%C3%89ric+de+la+Clergerie%2C+Djam%C3%A9+Seddah%2C+and+Beno%C3%AEt+Sagot%2E+2020%2E+CamemBERT%3A+a+tasty+French+language+model%2E+In+Proceed) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
39. Eneldo Loza Mencia and Johannes Fürnkranz. 2007. Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain. In Proceedings of the LWA 2007, pages 126–132. [find](https://scholar.google.com/scholar?q=Eneldo+Loza+Mencia+and+Johannes+F%C3%BCrnkranz%2E+2007%2E+Efficient+Multilabel+Classification+Algorithms+for+Large%2DScale+Problems+in+the+Legal+Domain%2E+In+Proceedings+of+the+LWA+2007%2C+pages+126%E2%80%93132%2E)
40. James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable Prediction of Medical Codes from Clinical Text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1101–1111. [find](https://scholar.google.com/scholar?q=James+Mullenbach%2C+Sarah+Wiegreffe%2C+Jon+Duke%2C+Jimeng+Sun%2C+and+Jacob+Eisenstein%2E+2018%2E+Explainable+Prediction+of+Medical+Codes+from+Clinical+Text%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Amer) [our card](https://md.research.elamur.ai/free/en/papers/mullenbach-2018-explainable-prediction-medical-codes-clinical/)
41. Dávid Márk Nemeskey. 2020. Natural Language Processing Methods for Language Modeling. Ph.D. thesis, Eötvös Loránd University. [find](https://scholar.google.com/scholar?q=D%C3%A1vid+M%C3%A1rk+Nemeskey%2E+2020%2E+Natural+Language+Processing+Methods+for+Language+Modeling%2E+Ph%2ED%2E+thesis%2C+E%C3%B6tv%C3%B6s+Lor%C3%A1nd+University%2E)
42. Son Nguyen, Le-Minh Nguyen, Satoshi Tojo, Ken Satoh, and Akira Shimazu. 2018. Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. Artificial Intelligence and Law, 26:1–31. [find](https://scholar.google.com/scholar?q=Son+Nguyen%2C+Le%2DMinh+Nguyen%2C+Satoshi+Tojo%2C+Ken+Satoh%2C+and+Akira+Shimazu%2E+2018%2E+Recurrent+neural+network%2Dbased+models+for+recognizing+requisite+and+effectuation+parts+in+legal+texts%2E+Artificial+Intellig)
43. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.eacl-main.39) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
44. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020%2E+MAD%2DX%3A+An+Adapter%2DBased+Framework+for+Multi%2DTask+Cross%2DLingual+Transfer%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
46. Amir Rosenfeld and John K. Tsotsos. 2019. Intriguing properties of randomly weighted networks: Generalizing while learning next to nothing. In 16th Conference on Computer and Robot Vision (CRV 2021), Kingston, ON, Canada. [find](https://scholar.google.com/scholar?q=Amir+Rosenfeld+and+John+K%2E+Tsotsos%2E+2019%2E+Intriguing+properties+of+randomly+weighted+networks%3A+Generalizing+while+learning+next+to+nothing%2E+In+16th+Conference+on+Computer+and+Robot+Vision+%28CRV+2021%29%2C+)
47. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: towards more challenging and nuanced multilingual evaluation. CoRR, abs/2104.07412. [↗](https://arxiv.org/abs/2104.07412) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/)
48. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2017. A survey of cross-lingual embedding models. CoRR, abs/1706.04902. [↗](https://arxiv.org/abs/1706.04902)
49. Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In 8th International Conference on Learning Representations (ICLR 2020). [find](https://scholar.google.com/scholar?q=Shiori+Sagawa%2C+Pang+Wei+Koh%2C+Tatsunori+B%2E+Hashimoto%2C+and+Percy+Liang%2E+2020%2E+Distributionally+robust+neural+networks+for+group+shifts%3A+On+the+importance+of+regularization+for+worst%2Dcase+generalization%2E)
50. Jakub Sido, Ondřej Pražák, Pavel Přibáň, Jan Pašek, Michal Seják, and Miloslav Konopík. 2021. Czert – czech bert-like model for language representation. arXiv preprint arXiv:2103.13031. [↗](https://arxiv.org/abs/2103.13031)
51. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems, pages 403– 417, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=F%C3%A1bio+Souza%2C+Rodrigo+Nogueira%2C+and+Roberto+Lotufo%2E+2020%2E+Bertimbau%3A+Pretrained+bert+models+for+brazilian+portuguese%2E+In+Intelligent+Systems%2C+pages+403%E2%80%93+417%2C+Cham%2E+Springer+International+Publishing%2E)
52. Felix Stahlberg. 2020. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2E+2020%2E+Neural+machine+translation%3A+A+review%2E+Journal+of+Artificial+Intelligence+Research%2C+69%2E)
53. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In IEEE conference on computer vision and pattern recognition. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+IEEE+conference+on+computer+vision+and+patt)
54. Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021. We need to talk about random splits. In Proceedings of the 2021 Conference of the European Chapter of the Association for Computational Linguistics (EACL), Online. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2C+Sebastian+Ebert%2C+Jasmijn+Bastings%2C+and+Katja+Filippova%2E+2021%2E+We+need+to+talk+about+random+splits%2E+In+Proceedings+of+the+2021+Conference+of+the+European+Chapter+of+the+Association+for+)
55. Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021. Revisiting the primacy of english in zero-shot cross-lingual transfer. CoRR, abs/2106.16171. [↗](https://arxiv.org/abs/2106.16171)
56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In 31th Annual Conference on Neural Information Processing Systems, USA. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+Is+All+You+Need%2E+In+31th+Annual+Conference+on+Neural+Infor)
57. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for finnish. CoRR, abs/1912.07076. [↗](https://arxiv.org/abs/1912.07076) [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/)
58. Bernhard Waltl, Johannes Muhr, Ingo Glaser, Georg Bonczek, Elena Scepankova, and Florian Matthes. 2017. Classifying legal norms with active machine learning. In 30th International Conference on Legal Knowledge and Information Systems (JURIX), pages 11–20, Luxembourg. [find](https://scholar.google.com/scholar?q=Bernhard+Waltl%2C+Johannes+Muhr%2C+Ingo+Glaser%2C+Georg+Bonczek%2C+Elena+Scepankova%2C+and+Florian+Matthes%2E+2017%2E+Classifying+legal+norms+with+active+machine+learning%2E+In+30th+International+Conference+on+Legal+)
59. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
60. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.naacl-main.41)
61. Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. CoRR, abs/2106.10199. [↗](https://arxiv.org/abs/2106.10199) [our card](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/)
62. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. [find](https://scholar.google.com/scholar?q=Haoxi+Zhong%2C+Chaojun+Xiao%2C+Cunchao+Tu%2C+Tianyang+Zhang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2020%2E+How+does+NLP+benefit+legal+system%3A+A+summary+of+legal+artificial+intelligence%2E+In+Proceedings+of+the+58th+Ann) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2020-how-does-nlp-benefit-legal/)