--- title: "Zeyuan Allen-Zhu — author profile and papers" description: "Zeyuan Allen-Zhu: 2 papers on Elamur Research (2021–2023) — affiliations, topics, co-authors and who cites them in our library." canonical: "https://research.elamur.ai/free/en/authors/zeyuan-allen-zhu/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/authors/zeyuan-allen-zhu/) [home](https://md.research.elamur.ai/free/en/) / [authors](https://md.research.elamur.ai/free/en/authors/) / Zeyuan Allen-Zhu # Zeyuan Allen-Zhu Zeyuan Allen-Zhu works on large language models. The papers here take them from two sides: how to adapt a pretrained model cheaply to new tasks, and what such a model actually learns from its training text. The methods range from changes to how Transformer weights are updated during fine-tuning to controlled experiments in which small models are trained from scratch on synthetic text, so that the influence of the training data can be isolated. Both papers belong to [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/), and one also to [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/). The most cited work is [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) (2021). Full fine-tuning updates every weight, which for a model the size of GPT-3 means keeping a separate full copy for each task. LoRA instead leaves the pretrained weights frozen. For selected Transformer layers it learns only the change to the weights, written as the product of two narrow low-rank matrices. This sidesteps the extra inference cost of adapter layers and the optimisation difficulties of prefix tuning. The more recent [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3/) (2023) asks whether a model genuinely extracts facts from its training text or merely memorises answers to familiar questions. To separate the two, GPT-2-style models are trained from scratch on a synthetic collection of biographies of fictional people. They are then fine-tuned on questions about some individuals and tested on the rest. The models can reproduce the biographies word for word. Without varied rephrasings of the same facts in the training text, however, they can hardly retrieve those facts for people held out from fine-tuning. Allen-Zhu's most frequent co-author in these papers is [Yuanzhi Li](https://md.research.elamur.ai/free/en/authors/yuanzhi-li/). Other collaborators include [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/), [Yelong Shen](https://md.research.elamur.ai/free/en/authors/yelong-shen/) and [Lu Wang](https://md.research.elamur.ai/free/en/authors/lu-wang/). Zeyuan Allen-Zhu has 2 papers in our library, in 2021–2023, most of them on [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/), [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/); often with [Yuanzhi Li](https://md.research.elamur.ai/free/en/authors/yuanzhi-li/), [Lu Wang](https://md.research.elamur.ai/free/en/authors/lu-wang/), [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/). 2 papers in our library · 2021–2023 No OpenAlex profile matched yet. [📄 2 works in our library](https://md.research.elamur.ai/free/en/authors/zeyuan-allen-zhu/#papers) · [🔗 6,065 citations in our library](https://md.research.elamur.ai/free/en/authors/zeyuan-allen-zhu/cited-by/) · [⭐ Paper Star rank #422](https://md.research.elamur.ai/free/en/paper-stars/page/5/#star-422) The profile is the library's match of the name; two people with one name are not always told apart. Elsewhere: [Google Scholar](https://scholar.google.com/citations?view_op=search_authors&mauthors=Zeyuan+Allen-Zhu) · [Semantic Scholar](https://www.semanticscholar.org/search?q=Zeyuan+Allen-Zhu) · [dblp](https://dblp.org/search?q=Zeyuan+Allen-Zhu) · [ACL Anthology](https://aclanthology.org/search/?q=Zeyuan+Allen-Zhu) ## Topics [Language models & pretraining (2)](https://md.research.elamur.ai/free/en/topics/language-models/) · [Efficient models & compression (1)](https://md.research.elamur.ai/free/en/topics/efficient-models/) ## Works by year ### 2020s [**2023** 1 paper](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3/) · [**2021** 1 paper](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) ## Papers The most cited first (OpenAlex and our library). 1. **[LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/)** *(2021 · 5,972 citations)* Large language models are usually adapted to specific tasks through full fine-tuning, that is, by updating all the weights. 2. **[Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3/)** *(2023 · 107 citations)* The authors test whether a language model actually learns to extract facts from the texts it was trained on or only memorizes answers to similar questions in the training data. ## Co-authors here 4 co-authors with a page, the most papers together first. - [Yuanzhi Li](https://md.research.elamur.ai/free/en/authors/yuanzhi-li/) ×2 - [Lu Wang](https://md.research.elamur.ai/free/en/authors/lu-wang/) - [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) - [Yelong Shen](https://md.research.elamur.ai/free/en/authors/yelong-shen/) ## Cited by Papers of our library cite their works 6,065 times (of the 140,884 papers whose bibliographies we read); [199 citing papers with a page here →](https://md.research.elamur.ai/free/en/authors/zeyuan-allen-zhu/cited-by/).