--- title: "LEGAL-BERT: The Muppets straight out of Law — Chalkidis 2020" description: "General-purpose BERT models are trained on text such as Wikipedia and books." authors: ["Ilias Chalkidis", "Manos Fergadiotis", "Prodromos Malakasiotis", "Nikolaos Aletras", "Ion Androutsopoulos"] year: 2020 venue: "Findings of the Association for Computational Linguistics: EMNLP 2020" source: "https://aclanthology.org/2020.findings-emnlp.261/" doi: "10.18653/v1/2020.findings-emnlp.261" arxiv: "2010.02559" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / LEGAL-BERT: The Muppets straight out of Law School # LEGAL-BERT: The Muppets straight out of Law School **Authors:** [Ilias Chalkidis](https://md.research.elamur.ai/free/en/authors/ilias-chalkidis/), [Manos Fergadiotis](https://md.research.elamur.ai/free/en/authors/manos-fergadiotis/), [Prodromos Malakasiotis](https://md.research.elamur.ai/free/en/authors/prodromos-malakasiotis/), [Nikolaos Aletras](https://md.research.elamur.ai/free/en/authors/nikolaos-aletras/), [Ion Androutsopoulos](https://md.research.elamur.ai/free/en/authors/ion-androutsopoulos/) · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.261/) · [PDF](https://aclanthology.org/2020.findings-emnlp.261.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.261](https://doi.org/10.18653/v1/2020.findings-emnlp.261) · [arXiv 2010.02559](https://arxiv.org/abs/2010.02559) · [OpenAlex](https://openalex.org/W3092129462) **Citations:** 994 citations (openalex-snapshot · 2026-10-14) · [496](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary General-purpose BERT models are trained on text such as Wikipedia and books. They tend to lose accuracy on specialised language, and legal text is a clear example, with its distinctive vocabulary, formal syntax and reliance on domain knowledge. The authors compare three ways of bringing BERT into the legal field: - using the original model unchanged; - continuing its pre-training on legal material (FP); - training a new model from scratch with its own sub-word vocabulary (SC). For the domain-specific variants, they assembled about 12 GB of English legal documents, including legislation, court decisions and contracts. They also tested a compact model and widened the hyper-parameter grid used for fine-tuning beyond the usual recommendations of Devlin et al. On three legal benchmarks, the domain-adapted models nearly always beat a carefully tuned BERT-BASE. The largest gains came on the hardest tasks, where in-domain knowledge matters most. The compact variant performed close to the full-size models at a fraction of the cost. The resulting LEGAL-BERT family is publicly released, giving legal NLP and legal-tech work ready-made domain models and practical advice for adapting BERT to other specialised fields. ## Key points - **Pre-training corpus:** six sources totalling about 12 GB. - EU and UK legislation - ECJ and ECHR cases - US court cases (Case Law Access Project) - US contracts (SEC-EDGAR) - **Models released:** LEGAL-BERT-FP, LEGAL-BERT-SC and LEGAL-BERT-SMALL, available at huggingface.co/nlpaueb. - LEGAL-BERT-FP continues training BERT-BASE for up to 500k steps. - LEGAL-BERT-SC has 110M parameters and a new sentencepiece vocabulary. - LEGAL-BERT-SMALL has 6 layers, 512 hidden units, 8 heads and 35M parameters, and runs roughly 4× faster. - **Training setup:** the from-scratch models were trained for 1M steps with batch size 256 and sequences of up to 512 tokens, on Cloud TPU v3. - **Evaluation datasets:** EURLEX57K (multi-label classification), ECHR-CASES (binary and multi-label classification) and CONTRACTS-NER (three subsets). - Gains were about +2.5% on ECHR multi-label, +1.8% on contract header and +1.6% on dispute resolution. - The gain on EURLEX57K was only about +0.2%. - **Fine-tuning:** an enlarged search space clearly improved results on most tasks. It added batch sizes 4–32, a learning rate of 1e-5, dropout of 0.2, and early stopping in place of a fixed epoch limit. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Law](https://md.research.elamur.ai/free/en/shelves/law/) ## Cited by Cited by 496 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 472 not yet. - [Automated Extraction of Sentencing Decisions from Court Cases in the Hebrew Language](https://md.research.elamur.ai/free/en/papers/aclbulk-2021-wenger-automated-extraction-sentencing-decisions-court/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [HateBERT: Retraining BERT for Abusive Language Detection in English](https://md.research.elamur.ai/free/en/papers/caselli-2021-hatebert-retraining-bert-abusive-language/) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/) - [MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/) - [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-paragraph-level-rationale-extraction-through/) - [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/) - [Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis](https://md.research.elamur.ai/free/en/papers/findings-2022-kulick-penn-helsinki-parsed-corpus-early/) - [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [Lawyer LLaMA Technical Report](https://md.research.elamur.ai/free/en/papers/huang-2023-lawyer-llama-technical-report/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-developing-pragmatic-benchmark-assessing-korean/) - [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/) - [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://md.research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/) - [Building Patient Journeys in Hebrew: A Language Model for Clinical Timeline Extraction](https://md.research.elamur.ai/free/en/papers/langstream-2025-hashiloni-building-patient-journeys-hebrew-language/) - [CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation](https://md.research.elamur.ai/free/en/papers/ma-2026-clase-hybrid-method-chinese-legalese/) - [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://md.research.elamur.ai/free/en/papers/malik-2021-ildc-cjpe-indian-legal-documents/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [Structured Legal Document Generation in India: A Model-Agnostic Wrapper Approach with VidhikDastaavej](https://md.research.elamur.ai/free/en/papers/nigam-2026-structured-legal-document-generation-india/) - [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://md.research.elamur.ai/free/en/papers/niklaus-2021-swiss-judgment-prediction-multilingual-legal/) - [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://md.research.elamur.ai/free/en/papers/niklaus-2023-lextreme-multi-lingual-multi-task/) - [AYN: A Tiny Yet Competitive Indian Legal Language Model Pretrained from Scratch](https://md.research.elamur.ai/free/en/papers/niyogi-2026-ayn-tiny-yet-competitive-indian/) - [CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering](https://md.research.elamur.ai/free/en/papers/wiratunga-2024-cbr-rag-case-based-reasoning/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{chalkidis2020legal, title = {{LEGAL-BERT: The Muppets straight out of Law School}}, author = {Chalkidis, Ilias and Fergadiotis, Manos and Malakasiotis, Prodromos and Aletras, Nikolaos and Androutsopoulos, Ion}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020}, year = {2020}, doi = {10.18653/v1/2020.findings-emnlp.261}, eprint = {2010.02559}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2020.findings-emnlp.261} } ``` - [← previous LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts](https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/) - [next → Lemmatising Verbs in Middle English Corpora: The Benefit of Enriching the Penn-Helsinki Parsed Corpus of Middle English 2 (PPCME2), the Parsed Corpus of Middle English Poetry (PCMEP), and A Parsed Linguistic Atlas of Early Middle English (PLAEME)](https://md.research.elamur.ai/free/en/papers/gram-2020-trips-lemmatising-verbs-middle-english-corpora/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) ## References 8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. I. Chalkidis, I. Androutsopoulos, and A. Michos. 2017. Extracting Contract Elements. In Proceedings of the International Conference of AI and Law, London, UK. [find](https://scholar.google.com/scholar?q=I%2E+Chalkidis%2C+I%2E+Androutsopoulos%2C+and+A%2E+Michos%2E+2017%2E+Extracting+Contract+Elements%2E+In+Proceedings+of+the+International+Conference+of+AI+and+Law%2C+London%2C+UK%2E) 2. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. [↗](https://doi.org/10.48550/arxiv.2001.04451) [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/) 3. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+RACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) 4. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. CoRR, abs/1909.11942. [↗](https://arxiv.org/abs/1909.11942) 5. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. In CoRR. [↗](https://doi.org/10.1093/bioinformatics/btz682) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) 6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692) 7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [↗](https://arxiv.org/abs/1606.05250) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 8. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies%2C+pages+1480%E2%80%931489%2E+Association+for+Computational+Linguisti)