---
title: "Robust Distortion-free Watermarks for Language Models (2023)"
description: "In the proposed protocol, the provider and detector share a secret key that encodes a sequence of… 30 references, cited by 93 papers here."
authors: ["Rohith Kuditipudi", "John Thickstun", "Tatsunori Hashimoto", "Percy Liang"]
year: 2023
venue: "arXiv:2307.15593"
source: "https://arxiv.org/abs/2307.15593"
doi: "10.48550/arxiv.2307.15593"
arxiv: "2307.15593"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kuditipudi-2023-robust-distortion-free-watermarks-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kuditipudi-2023-robust-distortion-free-watermarks-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Robust Distortion-free Watermarks for Language Models

# Robust Distortion-free Watermarks for Language Models

**Authors:** Rohith Kuditipudi, [John Thickstun](https://md.research.elamur.ai/free/en/authors/john-thickstun/), [Tatsunori Hashimoto](https://md.research.elamur.ai/free/en/authors/tatsunori-hashimoto/) and [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) · **Year:** 2023 · **Venue:** arXiv:2307.15593 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.15593) · [PDF](https://arxiv.org/pdf/2307.15593) · [DOI 10.48550/arxiv.2307.15593](https://doi.org/10.48550/arxiv.2307.15593) · [arXiv 2307.15593](https://arxiv.org/abs/2307.15593) · [OpenAlex](https://openalex.org/W4385437823)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [93](https://md.research.elamur.ai/free/en/papers/kuditipudi-2023-robust-distortion-free-watermarks-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles provenance of LM-generated text. It seeks a watermark that leaves the model's output distribution unchanged (distortion-free), can be verified without access to the model or prompt (agnostic), and survives editing (robust). Earlier hashing-based schemes (Kirchenbauer et al., Aaronson, Christ et al.) fall short on at least one of these. In the proposed protocol, the provider and detector share a secret key that encodes a sequence of random numbers. Generation deterministically turns this sequence into a model sample, using either inverse transform sampling (ITS) or exponential minimum sampling (EXP). A shift wrapper starts from a random offset so that outputs vary across calls. Detection aligns a suspect text against the key sequence using robust sequence-alignment costs, which yields a p-value. Experiments cover OPT-1.3B, LLaMA-7B and Alpaca-7B. The EXP variant is the strongest, staying detectable from short passages even after heavy random edits. Detection is harder on low-entropy instruction responses. The work matters because it shows watermarking need not trade output quality for robustness, and the code is publicly released.

## Key points
- **Robustness to edits:** On OPT-1.3B and LLaMA-7B, 35 tokens suffice for detection at p ≤ 0.01, even with 40–50% of tokens randomly substituted, inserted or deleted.
- **Robustness to translation:** Text paraphrased by round-trip translation through French or Russian remains detectable from about 50 tokens.
- **Alpaca-7B case study:** Only about 25% of instruction responses (median length around 100 tokens) are detectable at p ≤ 0.01, and robustness to automated paraphrasing is weaker.
- **Detection runtimes:** With key length n = 10,000 on an Apple M2 laptop, the test statistics take about 0.004 s (ITS), 0.60 s (ITS-edit), 2.20 s (EXP) and 3.21 s (EXP-edit).
- **Practical recommendations:** A small key (n = 256) does not noticeably harm text quality. EXP-edit is advised when robustness matters, and ITS when high-throughput detection matters, since it reduces to maximum inner-product search. Code is at github.com/jthickstun/watermark.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto and Percy Liang. 2023. Robust Distortion-free Watermarks for Language Models. arXiv:2307.15593.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kuditipudi2023robust,
  title = {{Robust Distortion-free Watermarks for Language Models}},
  author = {Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto and Percy Liang},
  year = {2023},
  doi = {10.48550/arxiv.2307.15593},
  eprint = {2307.15593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15593}
}
```

- [← previous GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP](https://md.research.elamur.ai/free/en/papers/khondaker-2023-gptaraeval-comprehensive-evaluation-chatgpt-arabic/)
- [next → Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://md.research.elamur.ai/free/en/papers/zhou-2023-solving-challenging-math-word-problems/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Aaronson. ‘Reform’ AI Alignment with Scott Aaronson. AXRP - the AI X-risk Research Podcast, 2023. URL [https://axrp.net/episode/2023/04/11/](https://axrp.net/episode/2023/04/11/) episode-20-reform-ai-alignment-scott-aaronson.html. [↗](https://axrp.net/episode/2023/04/11/)
2. S. Abdelnabi and M. Fritz. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In IEEE Symposium on Security and Privacy, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Abdelnabi+and+M%2E+Fritz%2E+Adversarial+watermarking+transformer%3A+Towards+tracing+text+provenance+with+data+hiding%2E+In+IEEE+Symposium+on+Security+and+Privacy%2C+2021%2E)
3. M. J. Atallah, V. Raskin, M. Crogan, C. Hempelmann, F. Kerschbaum, D. Mohamed, and S. Naik. Natural language watermarking: Design, analysis, and a proof-of-concept implementation. In Information Hiding: 4th International Workshop, IH 2001 Pittsburgh, PA, USA, April 25–27, 2001 Proceedings 4, pages 1… [find](https://scholar.google.com/scholar?q=M%2E+J%2E+Atallah%2C+V%2E+Raskin%2C+M%2E+Crogan%2C+C%2E+Hempelmann%2C+F%2E+Kerschbaum%2C+D%2E+Mohamed%2C+and+S%2E+Naik%2E+Natural+language+watermarking%3A+Design%2C+analysis%2C+and+a+proof%2Dof%2Dconcept+implementation%2E+In+Information+Hidin)
4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems 33, 33:1877–1901, 2020.
5. M. Christ, S. Gunn, and O. Zamir. Undetectable watermarks for language models. arXiv preprint arXiv:2306.09194, 2023. — [arXiv:2306.09194](https://arxiv.org/abs/2306.09194) [find](https://scholar.google.com/scholar?q=M%2E+Christ%2C+S%2E+Gunn%2C+and+O%2E+Zamir%2E+Undetectable+watermarks+for+language+models%2E+arXiv+preprint+arXiv%3A2306%2E09194%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E09194%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E09194%29)
6. F. Dai and Z. Cai. Towards near-imperceptible steganographic text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4303–4308, 2019. [find](https://scholar.google.com/scholar?q=F%2E+Dai+and+Z%2E+Cai%2E+Towards+near%2Dimperceptible+steganographic+text%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4303%E2%80%934308%2C+2019%2E)
7. G. Elfving, G. Birkhoff, and R. von Mises. Vol. 2. probability and statistics, General. In Selected Papers of Richard von Mises. American Mathematical Society, 1966. [find](https://scholar.google.com/scholar?q=G%2E+Elfving%2C+G%2E+Birkhoff%2C+and+R%2E+von+Mises%2E+Vol%2E+2%2E+probability+and+statistics%2C+General%2E+In+Selected+Papers+of+Richard+von+Mises%2E+American+Mathematical+Society%2C+1966%2E)
8. X. He, Q. Xu, L. Lyu, F. Wu, and C. Wang. Protecting intellectual property of language generation APIs with lexical watermark. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022. [find](https://scholar.google.com/scholar?q=X%2E+He%2C+Q%2E+Xu%2C+L%2E+Lyu%2C+F%2E+Wu%2C+and+C%2E+Wang%2E+Protecting+intellectual+property+of+language+generation+APIs+with+lexical+watermark%2E+In+Proceedings+of+the+Thirty%2DSixth+AAAI+Conference+on+Artificial+Intellig)
9. X. He, Q. Xu, Y. Zeng, L. Lyu, F. Wu, J. Li, and R. Jia. Cater: Intellectual property protection on text generation apis via conditional watermarks. In Advances in Neural Information Processing Systems 35, 2022. [find](https://scholar.google.com/scholar?q=X%2E+He%2C+Q%2E+Xu%2C+Y%2E+Zeng%2C+L%2E+Lyu%2C+F%2E+Wu%2C+J%2E+Li%2C+and+R%2E+Jia%2E+Cater%3A+Intellectual+property+protection+on+text+generation+apis+via+conditional+watermarks%2E+In+Advances+in+Neural+Information+Processing+System)
10. G. Jawahar, M. Abdul-Mageed, and V. Laks Lakshmanan. [Automatic detection of machine generated text: A critical survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/). In International Conference on Computational Linguistics, 2020.
11. N. S. Kamaruddin, A. Kamsin, L. Y. Por, and H. Rahman. A review of text watermarking: theory, methods, and applications. IEEE Access, 2018. [find](https://scholar.google.com/scholar?q=N%2E+S%2E+Kamaruddin%2C+A%2E+Kamsin%2C+L%2E+Y%2E+Por%2C+and+H%2E+Rahman%2E+A+review+of+text+watermarking%3A+theory%2C+methods%2C+and+applications%2E+IEEE+Access%2C+2018%2E)
12. S. Katzenbeisser and F. Petitcolas. Digital watermarking. Artech House, London, 2:2, 2000. [find](https://scholar.google.com/scholar?q=S%2E+Katzenbeisser+and+F%2E+Petitcolas%2E+Digital+watermarking%2E+Artech+House%2C+London%2C+2%3A2%2C+2000%2E)
13. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). arXiv preprint arXiv:2301.10226, 2023.
14. E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. [Detectgpt: Zero-shot machine-generated text detection using probability curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/). arXiv preprint arXiv:2301.11305, 2023.
15. G. Papandreou and A. L. Yuille. Perturb-and-map random fields: Using discrete optimization to learn and sample from energy models. 2011 International Conference on Computer Vision, pages 193–200, 2011. [find](https://scholar.google.com/scholar?q=G%2E+Papandreou+and+A%2E+L%2E+Yuille%2E+Perturb%2Dand%2Dmap+random+fields%3A+Using+discrete+optimization+to+learn+and+sample+from+energy+models%2E+2011+International+Conference+on+Computer+Vision%2C+pages+193%E2%80%93200%2C+2011)
16. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
17. S. G. Rizzo, F. Bertini, and D. Montesi. Fine-grain watermarking for intellectual property protection. EURASIP Journal on Information Security, 2019. [find](https://scholar.google.com/scholar?q=S%2E+G%2E+Rizzo%2C+F%2E+Bertini%2C+and+D%2E+Montesi%2E+Fine%2Dgrain+watermarking+for+intellectual+property+protection%2E+EURASIP+Journal+on+Information+Security%2C+2019%2E)
18. J. Shen, H. Ji, and J. Han. Near-imperceptible neural linguistic steganography via self-adjusting arithmetic coding. In Proceedings of Empirical Methods for Natural Language Processing, pages 303–313, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Shen%2C+H%2E+Ji%2C+and+J%2E+Han%2E+Near%2Dimperceptible+neural+linguistic+steganography+via+self%2Dadjusting+arithmetic+coding%2E+In+Proceedings+of+Empirical+Methods+for+Natural+Language+Processing%2C+pages+303%E2%80%93313%2C)
19. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
20. J. Tiedemann and S. Thottingal. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Tiedemann+and+S%2E+Thottingal%2E+OPUS%2DMT+%E2%80%94+Building+open+translation+services+for+the+World%2E+In+Proceedings+of+the+22nd+Annual+Conference+of+the+European+Association+for+Machine+Translation%2C+2020%2E)
21. J. Tiedemann, M. Aulamo, D. Bakshandaeva, M. Boggia, S.-A. Grönroos, T. Nieminen, A. Raganato, Y. Scherrer, R. Vazquez, and S. Virpioja. Democratizing machine translation with opus-mt. arXiv preprint arXiv:2212.01936, 2022. — [arXiv:2212.01936](https://arxiv.org/abs/2212.01936) [find](https://scholar.google.com/scholar?q=J%2E+Tiedemann%2C+M%2E+Aulamo%2C+D%2E+Bakshandaeva%2C+M%2E+Boggia%2C+S%2E%2DA%2E+Gro%CC%88nroos%2C+T%2E+Nieminen%2C+A%2E+Raganato%2C+Y%2E+Scherrer%2C+R%2E+Vazquez%2C+and+S%2E+Virpioja%2E+Democratizing+machine+translation+with+opus%2Dmt%2E+arXiv+preprint)
22. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
23. H. Ueoka, Y. Murawaki, and S. Kurohashi. Frustratingly easy edit-based linguistic steganography with a masked language model. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. [find](https://scholar.google.com/scholar?q=H%2E+Ueoka%2C+Y%2E+Murawaki%2C+and+S%2E+Kurohashi%2E+Frustratingly+easy+edit%2Dbased+linguistic+steganography+with+a+masked+language+model%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the)
24. A. Venugopal, J. Uszkoreit, D. Talbot, F. J. Och, and J. Ganitkevitch. Watermarking the outputs of structured prediction with an application in statistical machine translation. In Proceedings of Empirical Methods for Natural Language Processing, 2011. [find](https://scholar.google.com/scholar?q=A%2E+Venugopal%2C+J%2E+Uszkoreit%2C+D%2E+Talbot%2C+F%2E+J%2E+Och%2C+and+J%2E+Ganitkevitch%2E+Watermarking+the+outputs+of+structured+prediction+with+an+application+in+statistical+machine+translation%2E+In+Proceedings+of+Empir)
25. J. Vincent. AI-generated answers temporarily banned on coding Q&A site Stack Overflow. The Verge, 2022. URL [https://www.theverge.com/2022/12/5/23493932/](https://www.theverge.com/2022/12/5/23493932/) chatgpt-ai-generated-answers-temporarily-banned-stack-overflow-llms-dangers. [↗](https://www.theverge.com/2022/12/5/23493932/)
26. M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019. [find](https://scholar.google.com/scholar?q=M%2E+J%2E+Wainwright%2E+High%2DDimensional+Statistics%3A+A+Non%2DAsymptotic+Viewpoint%2E+Cambridge+University+Press%2C+2019%2E)
27. X. Yang, J. Zhang, K. Chen, W. Zhang, Z. Ma, F. Wang, and N. Yu. Tracing text provenance via context-aware lexical substitution. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Yang%2C+J%2E+Zhang%2C+K%2E+Chen%2C+W%2E+Zhang%2C+Z%2E+Ma%2C+F%2E+Wang%2C+and+N%2E+Yu%2E+Tracing+text+provenance+via+context%2Daware+lexical+substitution%2E+In+Proceedings+of+the+Thirty%2DSixth+AAAI+Conference+on+Artificial+Intell)
28. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.
29. X. Zhao, Y.-X. Wang, and L. Li. Protecting language generation models via invisible watermarking. arXiv preprint arXiv:2302.03162, 2023. — [arXiv:2302.03162](https://arxiv.org/abs/2302.03162) [find](https://scholar.google.com/scholar?q=X%2E+Zhao%2C+Y%2E%2DX%2E+Wang%2C+and+L%2E+Li%2E+Protecting+language+generation+models+via+invisible+watermarking%2E+arXiv+preprint+arXiv%3A2302%2E03162%2C+2023%2E+%E2%80%94+%5BarXiv%3A2302%2E03162%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E03162%29)
30. Z. Ziegler, Y. Deng, and A. M. Rush. Neural linguistic steganography. In Proceedings of Empirical Methods for Natural Language Processing, pages 1210–1215, 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Ziegler%2C+Y%2E+Deng%2C+and+A%2E+M%2E+Rush%2E+Neural+linguistic+steganography%2E+In+Proceedings+of+Empirical+Methods+for+Natural+Language+Processing%2C+pages+1210%E2%80%931215%2C+2019%2E)

*Updated 2 November 2026*