--- title: "LegalBench-RAG — paper by Pipitone and Alami (2024)" description: "RAG tools for law are spreading quickly, but existing legal benchmarks such as LegalBench only test… 1 reference, cited by 72 papers here." authors: ["Nicholas Pipitone", "Ghita Houir Alami"] year: 2024 venue: "arXiv:2408.10343" source: "https://arxiv.org/abs/2408.10343" doi: "10.48550/arxiv.2408.10343" arxiv: "2408.10343" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pipitone-2024-legalbench-rag-benchmark-retrieval-augmented/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pipitone-2024-legalbench-rag-benchmark-retrieval-augmented/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain # LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain **Authors:** Nicholas Pipitone and Ghita Houir Alami · **Year:** 2024 · **Venue:** arXiv:2408.10343 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.10343) · [PDF](https://arxiv.org/pdf/2408.10343) · [DOI 10.48550/arxiv.2408.10343](https://doi.org/10.48550/arxiv.2408.10343) · [arXiv 2408.10343](https://arxiv.org/abs/2408.10343) · [OpenAlex](https://openalex.org/W4403006857) **Citations:** 7 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/pipitone-2024-legalbench-rag-benchmark-retrieval-augmented/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary RAG tools for law are spreading quickly, but existing legal benchmarks such as LegalBench only test how well an LLM reasons when it is handed the correct passage. None of them check whether a system can find that passage in a large document collection. LegalBench-RAG addresses this gap. The authors took the clauses that LegalBench supplies as context and located each one in its original source document. Each answer is recorded as an exact character span. They then turned the source annotations into natural-language queries. The result is 6,858 expert-annotated query–span pairs over a corpus of more than 79M characters, drawn from four datasets. A smaller LegalBench-RAG-mini is also provided for quick experiments. In baseline tests, a Recursive Text Character Splitter beat naive chunking. Adding a general-purpose Cohere reranker unexpectedly lowered both precision and recall. The benchmark matters because legal answers need short, exact passages: short passages fit context limits, reduce hallucination, and let users check citations. ## Key points - The benchmark is built from four LegalBench sources: - ContractNLI (946 queries) - CUAD (4,042 queries) - MAUD (1,676 queries) - PrivacyQA (194 queries) - Ground truth for each query is a document file plus one or more character ranges, not a document ID or a loose chunk. - Queries follow the template "Consider (document description); (interrogative)". The description identifies the document, and the interrogative is mapped from the annotation category. - Recall@k and Precision@k were measured for k from 1 to 64, and the four datasets were weighted equally in the aggregate. - PrivacyQA turned out easiest: with RTCS and no reranker it reached about 14.4% Precision@1 and 84.2% Recall@64. MAUD was hardest, at about 2.6% Precision@1 and 28.3% Recall@64. ## Interesting ideas - **Convert a reader benchmark into a retrieval benchmark.** The authors traced each gold context passage back to its location in the full source documents. Any QA dataset with provided contexts can be repurposed this way, without new annotation work. - **Character-span ground truth.** Labelling answers as exact offsets, instead of document or chunk IDs, makes evaluation independent of how a system splits text. It also rewards precise snippets that fit in a context window and can be cited. - **Self-contained queries from templates.** Each query pairs a unique document description with a category question. This makes queries answerable across a whole corpus without hand-writing them, which is useful for building retrieval sets from classification-style annotations. - **Equal weighting per sub-dataset.** Averaging each source dataset equally, regardless of its size, keeps a large component like CUAD from dominating the score. This is a sensible default for any composite benchmark. - **Test general-purpose components on specialized text.** The finding that an off-the-shelf reranker hurt results on legal text argues for checking each pipeline stage on in-domain data rather than assuming it helps. This likely applies to other specialized domains as well. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 206 works 🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works · [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 150 works ## Cite Cite as: Nicholas Pipitone and Ghita Houir Alami. 2024. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. arXiv:2408.10343. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{pipitone2024legalbench, title = {{LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain}}, author = {Nicholas Pipitone and Ghita Houir Alami}, year = {2024}, doi = {10.48550/arxiv.2408.10343}, eprint = {2408.10343}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.10343} } ``` - [← previous Common 7B Language Models Already Possess Strong Math Capabilities](https://md.research.elamur.ai/free/en/papers/li-2024-common-7b-language-models-already/) - [next → Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/paul-2024-making-reasoning-matter-measuring-improving/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. [LexGLUE: A benchmark dataset for legal language understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2022-lexglue-benchmark-dataset-legal-language/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.acl-long.297. URL https://aclanthology.org/2022.acl-long.297. *Figure 5. Precision @ k across all datasets* [↗](https://aclanthology.org/2022.acl-long.297/) *Updated 2 November 2026*