---
title: "Ghostbuster — paper by Verma et al. (2024)"
description: "Ghostbuster tackles the detection of documents written by large language models when the… 99.0 F1. 28 references, cited by 70 papers here."
authors: ["Vivek Verma", "Eve Fleisig", "Nicholas Tomlin", "Dan Klein"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.95/"
doi: "10.18653/v1/2024.naacl-long.95"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/verma-2024-ghostbuster-detecting-text-ghostwritten-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/verma-2024-ghostbuster-detecting-text-ghostwritten-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Ghostbuster: Detecting Text Ghostwritten by Large Language Models

# Ghostbuster: Detecting Text Ghostwritten by Large Language Models

**Authors:** Vivek Verma, Eve Fleisig, [Nicholas Tomlin](https://md.research.elamur.ai/free/en/authors/nicholas-tomlin/) and [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.95/) · [PDF](https://aclanthology.org/2024.naacl-long.95.pdf) · [DOI 10.18653/v1/2024.naacl-long.95](https://doi.org/10.18653/v1/2024.naacl-long.95) · [OpenAlex](https://openalex.org/W4401042508)

**Citations:** 52 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/verma-2024-ghostbuster-detecting-text-ghostwritten-large/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Ghostbuster tackles the detection of documents written by large language models when the generating model is unknown or only reachable as a black box. Rather than requiring token probabilities from the suspected generator, it scores each text with several weaker language models: a unigram model, a trigram model, and the non-instruction-tuned GPT-3 davinci. A structured search then combines these per-token probabilities through vector and scalar operations into a compact feature set, and a logistic regression classifier is trained on the result. The authors also built three paired human/ChatGPT benchmarks covering creative writing, news, and student essays, with generated texts roughly matched in length to the human ones. Ghostbuster reaches 99.0 F1 across domains and generalizes better than DetectGPT, GPTZero, and a RoBERTa baseline to unseen domains, prompts, and generators such as Claude. The work matters because false accusations are costly, especially for students and non-native writers. It also shows that short texts and dedicated evasion services remain weak spots.

## Key points
- **Benchmark datasets:** Human texts come from r/WritingPrompts, the Reuters 50-50 authorship corpus, and IvyPanda essays. AI counterparts were produced with gpt-3.5-turbo. Claude outputs and alternative prompts were reserved for evaluation only.
- **Headline results:** Ghostbuster scores 99.0 F1, which is 5.9 F1 above the strongest prior detector. Its generalization gains are +7.5 F1 across domains, +2.1 F1 across prompts, and +4.4 F1 across models.
- **Ablations:** Dropping the neural LLM features lowers performance by up to 10.8 F1 in-domain and 27.8 F1 out-of-domain. Relying only on handcrafted features costs 15.9–22.1 F1.
- **Robustness:** The commercial evader Undetectable AI cuts recall from 99% to 62%. Detection is unreliable at ≤100 tokens and plateaus around 500 tokens.
- **Non-native writers:** Accuracy exceeds 95% on TOEFL 11 and Lang8. Results drop to 74.7 F1 on the set of 91 brief TOEFL essays collected by Liang et al. (2023), a decline the authors mostly blame on document length.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 127 works · 212,523 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Vivek Verma, Eve Fleisig, Nicholas Tomlin and Dan Klein. 2024. Ghostbuster: Detecting Text Ghostwritten by Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{verma2024ghostbuster,
  title = {{Ghostbuster: Detecting Text Ghostwritten by Large Language Models}},
  author = {Vivek Verma, Eve Fleisig, Nicholas Tomlin and Dan Klein},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.95},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.95}
}
```

- [← previous τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/)
- [next → InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://md.research.elamur.ai/free/en/papers/wang-2024-incharacter-evaluating-personality-fidelity-role/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Scott Aaronson. 2023. Watermarking of large language models. Workshop on Large Language Models and Transformers, Simons Institute, UC Berkeley. [find](https://scholar.google.com/scholar?q=Scott+Aaronson%2E+2023%2E+Watermarking+of+large+language+models%2E+Workshop+on+Large+Language+Models+and+Transformers%2C+Simons+Institute%2C+UC+Berkeley%2E)
2. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) [find](https://scholar.google.com/scholar?q=Anton+Bakhtin%2C+Sam+Gross%2C+Myle+Ott%2C+Yuntian+Deng%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Arthur+Szlam%2E+2019%2E+Real+or+fake%3F+learning+to+discriminate+machine+from+human+generated+text%2E+arXiv+preprint+arXiv%3A1906%2E0335)
3. Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. 2023. ConDA: Contrastive domain adaptation for AI-generated text detection. [find](https://scholar.google.com/scholar?q=Amrita+Bhattacharjee%2C+Tharindu+Kumarage%2C+Raha+Moraffah%2C+and+Huan+Liu%2E+2023%2E+ConDA%3A+Contrastive+domain+adaptation+for+AI%2Dgenerated+text+detection%2E)
4. Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. TOEFL11: A CORPUS OF NON-NATIVE EN-GLISH. ETS Research Report Series, 2013(2):i–15. [find](https://scholar.google.com/scholar?q=Daniel+Blanchard%2C+Joel+Tetreault%2C+Derrick+Higgins%2C+Aoife+Cahill%2C+and+Martin+Chodorow%2E+2013%2E+TOEFL11%3A+A+CORPUS+OF+NON%2DNATIVE+EN%2DGLISH%2E+ETS+Research+Report+Series%2C+2013%282%29%3Ai%E2%80%9315%2E)
5. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736. — [arXiv:2304.04736](https://arxiv.org/abs/2304.04736) [find](https://scholar.google.com/scholar?q=Souradip+Chakraborty%2C+Amrit+Singh+Bedi%2C+Sicheng+Zhu%2C+Bang+An%2C+Dinesh+Manocha%2C+and+Furong+Huang%2E+2023%2E+On+the+possibilities+of+ai%2Dgenerated+text+detection%2E+arXiv+preprint+arXiv%3A2304%2E04736%2E+%E2%80%94+%5BarXiv%3A230)
6. Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. 2023. GPT-Sentinel: Distinguishing human and ChatGPT generated content. [find](https://scholar.google.com/scholar?q=Yutian+Chen%2C+Hao+Kang%2C+Vivian+Zhai%2C+Liangze+Li%2C+Rita+Singh%2C+and+Bhiksha+Raj%2E+2023%2E+GPT%2DSentinel%3A+Distinguishing+human+and+ChatGPT+generated+content%2E)
7. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022. [Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text](https://md.research.elamur.ai/free/en/papers/dou-2022-is-gpt-3-text-indistinguishable/).
8. W Nelson Francis and Henry Kucera. 1979. Brown Corpus manual. Letters to the Editor, 5(2):7. [find](https://scholar.google.com/scholar?q=W+Nelson+Francis+and+Henry+Kucera%2E+1979%2E+Brown+Corpus+manual%2E+Letters+to+the+Editor%2C+5%282%29%3A7%2E)
9. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. 2019. [GLTR: Statistical detection and visualization of generated text](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/).
10. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. [How close is ChatGPT to human experts? comparison corpus, evaluation, and detection](https://md.research.elamur.ai/free/en/papers/guo-2023-close-chatgpt-human-experts-comparison/).
11. Will Heaven. 2023. ChatGPT is going to change education, not destroy it. MIT Technology Review. [find](https://scholar.google.com/scholar?q=Will+Heaven%2E+2023%2E+ChatGPT+is+going+to+change+education%2C+not+destroy+it%2E+MIT+Technology+Review%2E)
12. John Houvardas and Efstathios Stamatatos. 2006. Ngram feature selection for authorship identification. In Artificial Intelligence: Methodology, Systems, Applications. [find](https://scholar.google.com/scholar?q=John+Houvardas+and+Efstathios+Stamatatos%2E+2006%2E+Ngram+feature+selection+for+authorship+identification%2E+In+Artificial+Intelligence%3A+Methodology%2C+Systems%2C+Applications%2E)
13. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. [Automatic detection of generated text is easiest when humans are fooled](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/).
14. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks V. S. Lakshmanan. 2020. [Automatic detection of machine generated text: A critical survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/).
15. Nurul Shamimi Kamaruddin, Amirrudin Kamsin, Lip Yee Por, and Hameedur Rahman. 2018. A review of text watermarking: Theory, methods, and applications. IEEE Access, 6:8011–8028. [find](https://scholar.google.com/scholar?q=Nurul+Shamimi+Kamaruddin%2C+Amirrudin+Kamsin%2C+Lip+Yee+Por%2C+and+Hameedur+Rahman%2E+2018%2E+A+review+of+text+watermarking%3A+Theory%2C+methods%2C+and+applications%2E+IEEE+Access%2C+6%3A8011%E2%80%938028%2E)
16. Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. 2023. Provable robust watermarking for AI-generated text. arXiv preprint arXiv:2306.17439. [find](https://scholar.google.com/scholar?q=Xuandong+Zhao%2C+Prabhanjan+Ananth%2C+Lei+Li%2C+and+Yu%2DXiang+Wang%2E+2023%2E+Provable+robust+watermarking+for+AI%2Dgenerated+text%2E+arXiv+preprint+arXiv%3A2306%2E17439%2E)
17. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084. PMLR.
18. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. [Gpt detectors are biased against non-native english writers](https://md.research.elamur.ai/free/en/papers/liang-2023-gpt-detectors-biased-against-non/). Patterns, 4(7):100779.
19. Clara Meister and Ryan Cotterell. 2021. Language model evaluation beyond perplexity. [find](https://scholar.google.com/scholar?q=Clara+Meister+and+Ryan+Cotterell%2E+2021%2E+Language+model+evaluation+beyond+perplexity%2E)
20. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. [DetectGPT: Zero-shot machine-generated text detection using probability curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/).
21. Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–155, Chiang Mai,… [find](https://scholar.google.com/scholar?q=Tomoya+Mizumoto%2C+Mamoru+Komachi%2C+Masaaki+Nagata%2C+and+Yuji+Matsumoto%2E+2011%2E+Mining+revision+log+of+language+learning+SNS+for+automated+Japanese+error+correction+of+second+language+learners%2E+In+Proceedi)
22. OpenAI. 2019. GPT-2: 1.5b release. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2019%2E+GPT%2D2%3A+1%2E5b+release%2E)
23. Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhattacharya, Mobin Javed, and Bimal Viswanath. 2023. Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1613–1630. IEEE. [find](https://scholar.google.com/scholar?q=Jiameng+Pu%2C+Zain+Sarwar%2C+Sifat+Muhammad+Abdullah%2C+Abdullah+Rehman%2C+Yoonjin+Kim%2C+Parantapa+Bhattacharya%2C+Mobin+Javed%2C+and+Bimal+Viswanath%2E+2023%2E+Deepfake+text+detection%3A+Limitations+and+opportunities%2E+)
24. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. [Can AI-generated text be reliably detected?](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/)
25. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. [Release strategies and the social impacts of language models](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/).
26. Edward Tian. 2023. GPTZero: Home. [find](https://scholar.google.com/scholar?q=Edward+Tian%2E+2023%2E+GPTZero%3A+Home%2E)
27. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. [Authorship attribution for neural text generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384–8395, Online. Association for Computational Linguistics.
28. Vivek Verma, Nicholas Tomlin, and Dan Klein. 2023. Revisiting entropy rate constancy in text. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15537–15549, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vivek+Verma%2C+Nicholas+Tomlin%2C+and+Dan+Klein%2E+2023%2E+Revisiting+entropy+rate+constancy+in+text%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+pages+15537%E2%80%9315549%2C+Singapore%2E+As)

*Updated 24 October 2026*