--- title: "A Watermark for Large Language Models — Kirchenbauer 2023" description: "The paper tackles a practical question: how can text produced by a large language model be reliably distinguished from human writing? This matters…" authors: ["John Kirchenbauer", "Jonas Geiping", "Yuxin Wen", "Jonathan Katz", "Ian Miers", "Tom Goldstein"] year: 2023 venue: "arXiv:2301.10226" source: "https://arxiv.org/abs/2301.10226" doi: "10.48550/arxiv.2301.10226" arxiv: "2301.10226" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / A Watermark for Large Language Models # A Watermark for Large Language Models **Authors:** [John Kirchenbauer](https://md.research.elamur.ai/free/en/authors/john-kirchenbauer/), [Jonas Geiping](https://md.research.elamur.ai/free/en/authors/jonas-geiping/), [Yuxin Wen](https://md.research.elamur.ai/free/en/authors/yuxin-wen/), Jonathan Katz, Ian Miers, [Tom Goldstein](https://md.research.elamur.ai/free/en/authors/tom-goldstein/) · **Year:** 2023 · **Venue:** arXiv:2301.10226 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2301.10226) · [PDF](https://arxiv.org/pdf/2301.10226) · [DOI 10.48550/arxiv.2301.10226](https://doi.org/10.48550/arxiv.2301.10226) · [arXiv 2301.10226](https://arxiv.org/abs/2301.10226) · [OpenAlex](https://openalex.org/W4318149317) **Citations:** 110 citations (openalex-snapshot · 2026-10-14) · [263](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper tackles a practical question: how can text produced by a large language model be reliably distinguished from human writing? This matters because of misuse such as disinformation, social-media bots and academic cheating, and because synthetic content pollutes corpora used to train future models. Rather than building a post-hoc classifier, the authors hide a statistical signal in the text while it is generated. At every step, a hash of the preceding token seeds a random generator that splits the vocabulary into a "green" and a "red" list. In the soft variant, a constant δ is added to the logits of green tokens. This nudge changes the choice of tokens when many plausible options exist and barely affects nearly forced ones. To detect the watermark, one only needs the hash function: count the green tokens and run a one-proportion z-test, which yields interpretable p-values without loading the model or calling its API. The method is demonstrated on the OPT family. It requires no retraining, works on short spans of text, and makes false accusations against human authors statistically unlikely. That last property is important for groups like non-native speakers, whom existing detectors often wrongly flag. ## Key points - **Example result:** On OPT-6.7B with γ = 0.25 and δ = 2, a sample contained 28 green tokens where about 9 would be expected from a human. That corresponds to a chance probability of roughly 6×10⁻¹⁴. The watermark can be detected from spans as short as about 25 tokens. - **Hard red-list baseline:** Using a z > 4 threshold gives a false-positive rate of about 3×10⁻⁵. Any fully watermarked sequence of 16 or more tokens is caught. - **Robustness:** In a 1000-token text, an attacker who knows the scheme and adversarially changes 200 tokens still leaves z ≈ 6.3 (p ≈ 10⁻¹⁰). Erasing the mark generally requires editing roughly a quarter of the tokens. - **Detector independence:** The detector depends only on γ and the hashing scheme, not on δ. Generation settings can therefore be tuned per context (e.g., prose vs. code) while keeping a single detector. - **Code and demo:** Available at github.com/jwkirchenbauer/lm-watermarking. The detector can be released publicly or kept private behind an API. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 263 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 247 not yet. - [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://md.research.elamur.ai/free/en/papers/bao-2023-fast-detectgpt-efficient-zero-shot/) - [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/) - [Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) - [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://md.research.elamur.ai/free/en/papers/krishna-2023-paraphrasing-evades-detectors-ai-generated/) - [GPT detectors are biased against non-native English writers](https://md.research.elamur.ai/free/en/papers/liang-2023-gpt-detectors-biased-against-non/) - [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/) - [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://md.research.elamur.ai/free/en/papers/liu-2024-does-detectgpt-fully-utilize-perturbation/) - [Corpus and Baselines for Distinguishing Authentic, AI-Generated, and AI-Enhanced Resumes](https://md.research.elamur.ai/free/en/papers/loizidou-2026-corpus-baselines-distinguishing-authentic-ai/) - [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/) - [Can AI-Generated Text be Reliably Detected?](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/) - [SearchLLM: Detecting LLM Paraphrased Text by Measuring the Similarity with Regeneration of the Candidate Source via Search Engine](https://md.research.elamur.ai/free/en/papers/son-2026-searchllm-detecting-llm-paraphrased-text/) - [SWAN: Semantic Watermarking with Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/ye-2026-swan-semantic-watermarking-abstract-meaning/) - [The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks](https://md.research.elamur.ai/free/en/papers/zhao-2026-mark-fades-adaptive-evolutionary-paraphrase/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{kirchenbauer2023watermark, title = {{A Watermark for Large Language Models}}, author = {Kirchenbauer, John and Geiping, Jonas and Wen, Yuxin and Katz, Jonathan and Miers, Ian and Goldstein, Tom}, year = {2023}, doi = {10.48550/arxiv.2301.10226}, eprint = {2301.10226}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2301.10226} } ``` - [← previous A Transformer-based parser for Syriac morphology](https://md.research.elamur.ai/free/en/papers/aclbulk-2023-naaijer-transformer-based-parser-syriac-morphology/) - [next → Acquiring Linguistic Knowledge from Multimodal Input](https://md.research.elamur.ai/free/en/papers/amariucai-2023-acquiring-linguistic-knowledge-multimodal-input/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/) ## References 10 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Aaronson, S. My AI Safety Lecture for UT Effective Altruism, November 2022. URL [https://scottaaronson.blog/?p=6823](https://scottaaronson.blog/?p=6823). [↗](https://scottaaronson.blog/?p=6823) 2. Abdelnabi, S. and Fritz, M. Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 121–140, May 2021. doi: 10.1109/SP40001.2021.00083. — [doi:10.1109/sp40001.2021.00083](https://doi.org/10.1109/sp40001.2021.00083) [find](https://scholar.google.com/scholar?q=Abdelnabi%2C+S%2E+and+Fritz%2C+M%2E+Adversarial+Watermarking+Transformer%3A+Towards+Tracing+Text+Provenance+with+Data+Hiding%2E+In+2021+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pp%2E+121%E2%80%93140%2C+May+2021%2E+doi%3A+10%2E) 3. Adi, Y., Baum, C., Cisse, M., Pinkas, B., and Keshet, J. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In Proceedings of the 27th USENIX Conference on Security Symposium, SEC’18, pp. 1615–1631, USA, August 2018. USENIX Association. ISBN 978-1-931971-46-1. [find](https://scholar.google.com/scholar?q=Adi%2C+Y%2E%2C+Baum%2C+C%2E%2C+Cisse%2C+M%2E%2C+Pinkas%2C+B%2E%2C+and+Keshet%2C+J%2E+Turning+your+weakness+into+a+strength%3A+Watermarking+deep+neural+networks+by+backdooring%2E+In+Proceedings+of+the+27th+USENIX+Conference+on+Securi) 4. Atallah, M. J., Raskin, V., Crogan, M., Hempelmann, C., Kerschbaum, F., Mohamed, D., and Naik, S. Natural Language Watermarking: Design, Analysis, and a Proof-of-Concept Implementation. In Moskowitz, I. S. (ed.), Information Hiding, Lecture Notes in Computer Science, pp. 185–200, Berlin, Heidelberg,… — [doi:10.1007/3-540-45496-9](https://doi.org/10.1007/3-540-45496-9) [find](https://scholar.google.com/scholar?q=Atallah%2C+M%2E+J%2E%2C+Raskin%2C+V%2E%2C+Crogan%2C+M%2E%2C+Hempelmann%2C+C%2E%2C+Kerschbaum%2C+F%2E%2C+Mohamed%2C+D%2E%2C+and+Naik%2C+S%2E+Natural+Language+Watermarking%3A+Design%2C+Analysis%2C+and+a+Proof%2Dof%2DConcept+Implementation%2E+In+Moskowitz%2C+) 5. Atallah, M. J., Raskin, V., Hempelmann, C. F., Karahan, M., Sion, R., Topkara, U., and Triezenberg, K. E. Natural Language Watermarking and Tamperproofing. In Petitcolas, F. A. P. (ed.), Information Hiding, Lecture Notes in Computer Science, pp. 196–212, Berlin, Heidelberg, 2003. Springer. ISBN 978-… — [doi:10.1007/3-540-36415-3](https://doi.org/10.1007/3-540-36415-3) [find](https://scholar.google.com/scholar?q=Atallah%2C+M%2E+J%2E%2C+Raskin%2C+V%2E%2C+Hempelmann%2C+C%2E+F%2E%2C+Karahan%2C+M%2E%2C+Sion%2C+R%2E%2C+Topkara%2C+U%2E%2C+and+Triezenberg%2C+K%2E+E%2E+Natural+Language+Watermarking+and+Tamperproofing%2E+In+Petitcolas%2C+F%2E+A%2E+P%2E+%28ed%2E%29%2C+Information+H) 6. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosiute,… [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Kadavath%2C+S%2E%2C+Kundu%2C+S%2E%2C+Askell%2C+A%2E%2C+Kernion%2C+J%2E%2C+Jones%2C+A%2E%2C+Chen%2C+A%2E%2C+Goldie%2C+A%2E%2C+Mirhoseini%2C+A%2E%2C+McKinnon%2C+C%2E%2C+Chen%2C+C%2E%2C+Olsson%2C+C%2E%2C+Olah%2C+C%2E%2C+Hernandez%2C+D%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Li%2C+D%2E%2C) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/) 7. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610– 623, New York, NY, USA, March 2021. Association for Comp… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) [find](https://scholar.google.com/scholar?q=Bender%2C+E%2E+M%2E%2C+Gebru%2C+T%2E%2C+McMillan%2DMajor%2C+A%2E%2C+and+Shmitchell%2C+S%2E+On+the+Dangers+of+Stochastic+Parrots%3A+Can+Language+Models+Be+Too+Big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountabi) 8. Bergman, A. S., Abercrombie, G., Spruit, S., Hovy, D., Dinan, E., Boureau, Y.-L., and Rieser, V. Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 39–52, Edinburgh,… [find](https://scholar.google.com/scholar?q=Bergman%2C+A%2E+S%2E%2C+Abercrombie%2C+G%2E%2C+Spruit%2C+S%2E%2C+Hovy%2C+D%2E%2C+Dinan%2C+E%2E%2C+Boureau%2C+Y%2E%2DL%2E%2C+and+Rieser%2C+V%2E+Guiding+the+Release+of+Safer+E2E+Conversational+AI+through+Value+Sensitive+Design%2E+In+Proceedings+of+th) 9. Boenisch, F. A Systematic Review on Model Watermarking for Neural Networks. Frontiers in Big Data, 4, 2021. ISSN 2624-909X. doi: 10.3389/fdata.2021.729663. URL [https://www.frontiersin.org/articles/10.3389/fdata.2021.729663](https://www.frontiersin.org/articles/10.3389/fdata.2021.729663). — [doi:10.3389/fdata.2021.729663](https://doi.org/10.3389/fdata.2021.729663) [↗](https://www.frontiersin.org/articles/10.3389/fdata.2021.729663) 10. Boucher, N., Shumailov, I., Anderson, R., and Papernot, N. Bad Characters: Imperceptible NLP Attacks. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1987– 2004, May 2022. doi: 10.1109/SP46214.2022.9833641. — [doi:10.1109/sp46214.2022.9833641](https://doi.org/10.1109/sp46214.2022.9833641) [find](https://scholar.google.com/scholar?q=Boucher%2C+N%2E%2C+Shumailov%2C+I%2E%2C+Anderson%2C+R%2E%2C+and+Papernot%2C+N%2E+Bad+Characters%3A+Imperceptible+NLP+Attacks%2E+In+2022+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pp%2E+1987%E2%80%93+2004%2C+May+2022%2E+doi%3A+10%2E1109%2FSP4621)