--- title: "Large Language Models as Tax Attorneys: A Case Study in Legal" description: "The authors wrote Python generators that create new multiple-choice problems with random facts, names and… Cited by 37 papers here." authors: ["John J. Nay", "David Karamardian", "Sarah B. Lawsky", "Wenting Tao", "Meghana Bhat", "Raghav Jain", "Aaron Travis Lee", "Jonathan H. Choi", "Jungo Kasai"] year: 2023 venue: "arXiv:2306.07075" source: "https://arxiv.org/abs/2306.07075" doi: "10.48550/arxiv.2306.07075" arxiv: "2306.07075" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/nay-2023-large-language-models-tax-attorneys/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nay-2023-large-language-models-tax-attorneys/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence # Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence **Authors:** [John J. Nay](https://md.research.elamur.ai/free/en/authors/john-j-nay/), David Karamardian, [Sarah B. Lawsky](https://md.research.elamur.ai/free/en/authors/sarah-b-lawsky/), Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi and [Jungo Kasai](https://md.research.elamur.ai/free/en/authors/jungo-kasai/) · **Year:** 2023 · **Venue:** arXiv:2306.07075 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2306.07075) · [PDF](https://arxiv.org/pdf/2306.07075) · [DOI 10.48550/arxiv.2306.07075](https://doi.org/10.48550/arxiv.2306.07075) · [arXiv 2306.07075](https://arxiv.org/abs/2306.07075) · [OpenAlex](https://openalex.org/W4380558781) **Citations:** 12 citations (OpenAlex, 2026-10-14) · [37](https://md.research.elamur.ai/free/en/papers/nay-2023-large-language-models-tax-attorneys/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper asks how well large language models can apply law, using U.S. federal tax law as a testbed. Tax suits this purpose because its authority sits mainly in two fixed sources: the U.S. Code's Title 26 and the Treasury Regulations contained in the CFR. Many tax questions also have one verifiable answer that requires arithmetic as well as reading. The authors wrote Python generators that create new multiple-choice problems with random facts, names and figures, so no test item could have leaked into a training corpus. They tested successive OpenAI models, up to GPT-4. Each model ran alone, with legal text supplied through several retrieval approaches, and with few-shot examples. GPT-4 also served as the automatic grader. Accuracy improved with every newer model release. GPT-4 gained noticeably from few-shot examples and from receiving the correct source passages, but it still did not reach the level of an expert tax lawyer. The study offers a contamination-resistant way to track emerging legal reasoning in LLMs. ## Key points - **Two synthetic exams.** The CFR exam has three question types and the U.S. Code exam has four, with 100 questions per type. - **What each generated item contains.** Every item includes answer options, the correct choice, an explanation, and a citation to the governing provision. - **Topics covered.** Questions span "check the box" entity classification, restricted and unrestricted property as compensation, part gift/part sale, basis of gifted or inherited property, and the Section 199A qualified business income deduction. - **Automated grading.** More than 28,000 answers across all experiments were scored automatically by GPT-4, which compared each model's output against the ground truth. - **Best-performing setup.** The strongest configuration (the "mega_run" prompt) gave GPT-4 the relevant ("gold truth") legal sources, three solved examples, and a step-by-step reasoning cue. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 206 works 🧭 Topics (1): [⚖️ Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) · 150 works ## Cite Cite as: John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi and Jungo Kasai. 2023. Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence. arXiv:2306.07075. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{nay2023large, title = {{Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence}}, author = {John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi and Jungo Kasai}, year = {2023}, doi = {10.48550/arxiv.2306.07075}, eprint = {2306.07075}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2306.07075} } ``` - [← previous How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech](https://md.research.elamur.ai/free/en/papers/yedetore-2023-how-poor-is-stimulus-evaluating/) - [next → GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning](https://md.research.elamur.ai/free/en/papers/fan-2023-grammargpt-exploring-open-source-llms/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) *Updated 2 November 2026*