---
title: "LeXFiles and LegalLAMA — paper by Chalkidis et al. (2023)"
description: "Legal NLP already has many corpora, datasets and domain-adapted language models. 34 references, cited by 49 papers here."
authors: ["Ilias Chalkidis", "Nicolas Garneau", "Catalina Goanta", "Daniel Katz", "Anders Søgaard"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.865/"
doi: "10.18653/v1/2023.acl-long.865"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chalkidis-2023-lexfiles-legallama-facilitating-english-multinational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chalkidis-2023-lexfiles-legallama-facilitating-english-multinational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

# LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

**Authors:** [Ilias Chalkidis](https://md.research.elamur.ai/free/en/authors/ilias-chalkidis/), Nicolas Garneau, Catalina Goanta, [Daniel Katz](https://md.research.elamur.ai/free/en/authors/daniel-martin-katz/) and [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.865/) · [PDF](https://aclanthology.org/2023.acl-long.865.pdf) · [DOI 10.18653/v1/2023.acl-long.865](https://doi.org/10.18653/v1/2023.acl-long.865) · [OpenAlex](https://openalex.org/W4385571145)

**Citations:** 34 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/chalkidis-2023-lexfiles-legallama-facilitating-english-multinational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Legal NLP already has many corpora, datasets and domain-adapted language models. What has been missing is a systematic look at how pre-training data and model size shape what such models know and how well they perform, and at how far that knowledge stays tied to particular legal systems. The authors assemble LeXFiles, an English-language corpus covering six jurisdictions, most of them anglophone: India, the UK, the US, Canada, the EU and the CoE. On its training split they continue pre-training RoBERTa to produce two LexLM models. They also create LegalLAMA, a zero-shot cloze-style probing suite that targets legal knowledge specific to each jurisdiction.

Seven PLMs are compared on three levels:
- **Upstream:** MLM performance on each LeXFiles sub-corpus.
- **Probing:** LegalLAMA.
- **Downstream:** fine-tuning on LexGLUE tasks.

The main findings:
- Probing scores track MLM quality on related legal topics.
- For upstream and probing results, the choice of pre-training corpus matters more than model capacity.
- Downstream results depend mostly on model size plus the prior legal knowledge that the first two levels reveal.

The message for anyone building domain-specific models is that both data diversity and scale count.

## Key points
- LeXFiles has 11 sub-corpora (legislation, case law and US contracts) with about 5.8M documents and about 18.8B tokens. US case law alone is 59.2% of the tokens, so training uses smoothed sampling across sub-corpora.
- LeXFiles is far less US-centred than Pile of Law (32B tokens, where 26 of 30 sub-corpora are US). US case law is limited to decisions from 1965 onward, to reduce reliance on outdated, discriminatory legal standards.
- The two LexLM models are warm-started from RoBERTa and trained for 1M extra steps. They are encoder-only, and the models studied go up to about 350M parameters.
- LegalLAMA has 8 sub-tasks, scored with macro-averaged MRR over the candidate label set. Examples include:
  - ECHR Articles: 5,072 test instances, 13 article numbers.
  - Contractual Section Titles (US): 1,527 instances, 20 titles.
  - Contract Types (US): 1,089 instances, 15 types.
- LegalLAMA examples are taken from the LeXFiles test split. All models are fine-tuned for only one epoch on LexGLUE, to expose how prior knowledge affects convergence.

## Interesting ideas
- Measure a model at three linked levels (raw training objective, targeted knowledge probes, end-task results) and then correlate them, so you can see which cheap signal predicts the expensive one.
- Score cloze probes by rank only within a closed set of plausible labels rather than the whole vocabulary. This removes noise from irrelevant tokens and lets near-misses earn partial credit through reciprocal rank.
- When a target term splits into several subword pieces, mask each piece and average its reciprocal rank. This is a lenient but workable way to evaluate multi-token answers.
- Use smoothed sampling across very unequal sub-corpora so one dominant source does not drown out the smaller ones during training.
- Fine-tune deliberately briefly (a single epoch), so that results reflect what the model already knew rather than what heavy task training can force in.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Track parser loss, focused probe sets and downstream answer accuracy side by side, to learn which intermediate metric reliably forecasts gains in the readers.
- **I2** (world): Probe the compiled knowledge (verb classes, ontology relations) with masked sentences restricted to a small candidate set, giving a graded and explainable knowledge score.
- **I3** (tokens): When checking tagger or lemmatizer behaviour on multiword expressions, score each component separately and average, instead of all-or-nothing matching.
- **I4** (tree): When mixing treebanks of very different sizes and UD dialects to train gparse, use smoothed sampling rates to keep small genres represented.
- **I5** (learn): Run short, fixed-budget training as a control when comparing silver-data variants, to separate data quality from training length.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 115 works · 15,292 citations
- [Utrecht University](https://md.research.elamur.ai/free/en/universities/utrecht-university/) — University, Utrecht, Netherlands · 39 works · 1,603 citations
- [Illinois Institute of Technology](https://md.research.elamur.ai/free/en/universities/illinois-institute-of-technology/) — University, Chicago, United States · 12 works · 1,006 citations
- [Chicago Kent College of Law](https://md.research.elamur.ai/free/en/universities/chicago-kent-college-of-law/) — University, Chicago, United States · 2 works · 322 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 134 works

## Cite

Cite as: Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz and Anders Søgaard. 2023. LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chalkidis2023lexfiles,
  title = {{LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development}},
  author = {Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Katz and Anders Søgaard},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.865},
  url = {https://doi.org/10.18653/v1/2023.acl-long.865}
}
```

- [← previous DOC: Improving Long Story Coherence With Detailed Outline Control](https://md.research.elamur.ai/free/en/papers/yang-2023-doc-improving-long-story-coherence/)
- [next → ORCA: A Challenging Benchmark for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

34 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150. [↗](https://arxiv.org/abs/2004.05150)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
3. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021a. [MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2109.00904)
4. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. [LEGAL-BERT: The muppets straight out of law school](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online.
5. Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021b. [Paragraph-level rationale extraction through regularization: A case study on European court of human rights cases](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-paragraph-level-rationale-extraction-through/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 226–241, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2103.13084)
6. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022a. [LexGLUE: A benchmark dataset for legal language understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.2139/ssrn.3936759)
7. Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Schwemer, and Anders Søgaard. 2022b. FairLex: A multilingual benchmark for evaluating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389–4406, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Tommaso+Pasini%2C+Sheng+Zhang%2C+Letizia+Tomada%2C+Sebastian+Schwemer%2C+and+Anders+S%C3%B8gaard%2E+2022b%2E+FairLex%3A+A+multilingual+benchmark+for+evaluating+fairness+in+legal+text+processing%2E+In+Proc)
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). [↗](https://doi.org/10.48550/arxiv.2204.02311)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. [↗](https://arxiv.org/abs/1911.02116)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
11. Lawrence M. Friedman and Grant M. Hayden. 2017. 1What Is a Legal System? In American Law: An Introduction. Oxford University Press. [find](https://scholar.google.com/scholar?q=Lawrence+M%2E+Friedman+and+Grant+M%2E+Hayden%2E+2017%2E+1What+Is+a+Legal+System%3F+In+American+Law%3A+An+Introduction%2E+Oxford+University+Press%2E)
12. Ivan Habernal, Daniel Faber, Nicola Recchia, Sebastian Bretthauer, Iryna Gurevych, Indra Spiecker genannt Döhmann, and Christoph Burchard. 2022. Mining Legal Arguments in Court Decisions. arXiv preprint. [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Daniel+Faber%2C+Nicola+Recchia%2C+Sebastian+Bretthauer%2C+Iryna+Gurevych%2C+Indra+Spiecker+genannt+D%C3%B6hmann%2C+and+Christoph+Burchard%2E+2022%2E+Mining+Legal+Arguments+in+Court+Decisions%2E+arXiv+prepri)
13. Peter Henderson*, Mark S. Krass*, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. [find](https://scholar.google.com/scholar?q=Peter+Henderson%2A%2C+Mark+S%2E+Krass%2A%2C+Lucia+Zheng%2C+Neel+Guha%2C+Christopher+D%2E+Manning%2C+Dan+Jurafsky%2C+and+Daniel+E%2E+Ho%2E+2022%2E+Pile+of+law%3A+Learning+responsible+data+filtering+from+the+law+and+a+256gb+open%2Ds)
14. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/).
15. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
16. Wonseok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lee, and Minjoon Seo. 2022. A multi-task benchmark for korean legal language understanding and judgement prediction. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Wonseok+Hwang%2C+Dongjun+Lee%2C+Kyoungyeon+Cho%2C+Hanuhl+Lee%2C+and+Minjoon+Seo%2E+2022%2E+A+multi%2Dtask+benchmark+for+korean+legal+language+understanding+and+judgement+prediction%2E+In+Thirty%2Dsixth+Conference+on+Ne)
17. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361)
18. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam. [↗](https://doi.org/10.1098/rsta.2023.0254)
19. Yuta Koreeda and Christopher Manning. 2021. [ContractNLI: A dataset for document-level natural language inference for contracts](https://md.research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919, Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48448/3f52-xq43)
20. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf. 2021. [Datasets: A community library for natural language processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/). [↗](https://doi.org/10.48550/arxiv.2109.02846)
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
22. Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. [ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation](https://md.research.elamur.ai/free/en/papers/malik-2021-ildc-cjpe-indian-legal-documents/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4046–4062, Online. Association for Computational Linguistics.
23. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023. [Lextreme: A multi-lingual and multi-task benchmark for the legal domain](https://md.research.elamur.ai/free/en/papers/niklaus-2023-lextreme-multi-lingual-multi-task/). [↗](https://doi.org/10.24451/arbor.22335)
24. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186– 10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
25. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford+and+Karthik+Narasimhan%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
27. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. [AdapterDrop: On the efficiency of adapters in transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930–7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
28. Magnus Sahlgren and Fredrik Carlsson. 2021. The singleton fallacy: Why current critiques of language models miss the point. Frontiers in Artificial Intelligence, 4. [↗](https://doi.org/10.3389/frai.2021.682578)
29. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+In+Advances+in+Neu)
30. Dimitrios Tsarapatsanis and Nikolaos Aletras. 2021. On the ethical limits of natural language processing on legal text. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3590–3599, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dimitrios+Tsarapatsanis+and+Nikolaos+Aletras%2E+2021%2E+On+the+ethical+limits+of+natural+language+processing+on+legal+text%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+p)
31. Don Tuggener, Pius von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. [LEDGAR: A large-scale multi-label corpus for text classification of legal provisions in contracts](https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1235–1241, Marseille, France. European Language Resources Association.
32. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d19-1250)
33. Ellen Voorhees and D Tice. 2000. [The trec-8 question answering track](https://md.research.elamur.ai/free/en/papers/voorhees-2000-trec-8-question-answering-track/) evaluation. 3. The TREC-8 Question Answering Track Evaluation.
34. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Pro-Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In Proceedings of Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). CoRR, abs/2109.01652. [↗](https://arxiv.org/abs/2109.01652)

*Updated 1 November 2026*