---
title: "DetectLLM: Leveraging Log Rank Information for — Su 2023"
description: "As LLMs improve, telling their output apart from human writing gets harder, and supervised detectors must be retrained for every new model."
authors: ["Jinyan Su", "Terry Zhuo", "Di Wang", "Preslav Nakov"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.827/"
doi: "10.18653/v1/2023.findings-emnlp.827"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/su-2023-detectllm-leveraging-log-rank-information/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/su-2023-detectllm-leveraging-log-rank-information/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text

# DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text

**Authors:** Jinyan Su, Terry Zhuo, [Di Wang](https://md.research.elamur.ai/free/en/authors/di-wang/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.827/) · [PDF](https://aclanthology.org/2023.findings-emnlp.827.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.827](https://doi.org/10.18653/v1/2023.findings-emnlp.827) · [OpenAlex](https://openalex.org/W4389518647)

**Citations:** 75 citations (openalex-snapshot · 2026-10-14) · [97](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
As LLMs improve, telling their output apart from human writing gets harder, and supervised detectors must be retrained for every new model. This paper addresses zero-shot detection in a white-box setting: the detector can query the source model's token probabilities and ranks, but no training is involved. The authors propose two scoring features based on token log-rank. DetectLLM-LRR divides a text's negated average log-likelihood by its average log-rank, so it needs only a single pass over the text. DetectLLM-NPR compares the log-rank of slightly rewritten versions of a text, produced with a T5 model, against the log-rank of the original. It relies on the observation that machine-generated text reacts more strongly to such small edits. The methods were evaluated on XSum, SQuAD and WritingPrompts with seven LLMs from 1.5B to 20B parameters. Both outperform the strongest earlier zero-shot methods in their category, measured by AUROC. NPR also matches DetectGPT with fewer perturbations. The work gives cheaper detectors that need no retraining, along with practical advice on trading speed against accuracy.

## Key points
- On average, the two methods improve on the previous best zero-shot detectors by 3.9 and 1.75 absolute AUROC points.
- Each experiment uses 300 human/machine text pairs. The machine texts are generated by prompting the LLM with the first 30 tokens of the human text, following the DetectGPT protocol.
- Perturbation guidance:
  - T5-3b gives a higher performance ceiling than T5-large.
  - T5-3b usually needs fewer than 10 perturbations for NPR to match LRR.
  - T5-large needs more than twice as many perturbations for equal performance.
  - If only T5-small or T5-base is affordable, LRR is the better choice.
- To choose between the two methods, the authors suggest running NPR with 10 perturbations on a validation set and comparing it with LRR.
- The main limitations are the need for white-box access to the source model and English-only experiments. Weak supervision is proposed as future work.
- Code and data are available at github.com/mbzuai-nlp/DetectLLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 92 not yet.

- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://md.research.elamur.ai/free/en/papers/bao-2023-fast-detectgpt-efficient-zero-shot/)
- [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/)
- [GhostWriter: Hidden AI-Generated Texts over Multiple Languages, Domains and Generators](https://md.research.elamur.ai/free/en/papers/schaaf-2026-ghostwriter-hidden-ai-generated-texts/)
- [SearchLLM: Detecting LLM Paraphrased Text by Measuring the Similarity with Regeneration of the Candidate Source via Search Engine](https://md.research.elamur.ai/free/en/papers/son-2026-searchllm-detecting-llm-paraphrased-text/)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://md.research.elamur.ai/free/en/papers/wu-2025-survey-llm-generated-text-detection/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{su2023detectllm,
  title = {{DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text}},
  author = {Su, Jinyan and Zhuo, Terry and Wang, Di and Nakov, Preslav},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.827},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.827}
}
```

- [← previous Detecting Suicidality in Arabic Tweets Using Machine Learning and Deep Learning Techniques](https://md.research.elamur.ai/free/en/papers/abdulsalam-2023-detecting-suicidality-arabic-tweets-using/)
- [next → Diacritic Recognition Performance in Arabic ASR](https://md.research.elamur.ai/free/en/papers/aldarmaki-2023-diacritic-recognition-performance-arabic-asr/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

41 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sahar Abdelnabi and Mario Fritz. 2021. Adversarial watermarking transformer: Towards tracing text provenance with data hiding. In 2021 IEEE Symposium on Security and Privacy (SP), pages 121–140. IEEE. [find](https://scholar.google.com/scholar?q=Sahar+Abdelnabi+and+Mario+Fritz%2E+2021%2E+Adversarial+watermarking+transformer%3A+Towards+tracing+text+provenance+with+data+hiding%2E+In+2021+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pages+121%E2%80%93140%2E+IEEE%2E)
2. Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. 2008. Identifying real or fake articles: Towards better language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II. [find](https://scholar.google.com/scholar?q=Sameer+Badaskar%2C+Sachin+Agarwal%2C+and+Shilpa+Arora%2E+2008%2E+Identifying+real+or+fake+articles%3A+Towards+better+language+modeling%2E+In+Proceedings+of+the+Third+International+Joint+Conference+on+Natural+Lang)
3. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) [find](https://scholar.google.com/scholar?q=Anton+Bakhtin%2C+Sam+Gross%2C+Myle+Ott%2C+Yuntian+Deng%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Arthur+Szlam%2E+2019%2E+Real+or+fake%3F+learning+to+discriminate+machine+from+human+generated+text%2E+arXiv+preprint+arXiv%3A1906%2E0335)
4. Daria Beresneva. 2016. Computer-generated text detection using machine learning: A systematic review. In Natural Language Processing and Information Systems: 21st International Conference on Applications of Natural Language to Information Systems, NLDB 2016, Salford, UK, June 22-24, 2016, Proceeding… [find](https://scholar.google.com/scholar?q=Daria+Beresneva%2E+2016%2E+Computer%2Dgenerated+text+detection+using+machine+learning%3A+A+systematic+review%2E+In+Natural+Language+Processing+and+Information+Systems%3A+21st+International+Conference+on+Applicati)
5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregr… [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Stella+Biderman%2C+Eric+Hallahan%2C+Quentin+Anthony%2C+Leo+Gao%2C+Laurence+Golding%2C+Horace+He%2C+Connor+Leahy%2C+Kyle+McDonell%2C+Jason+Phang%2C+Michael+Pieler%2C+USVSN+Sai+Prashanth%2C+Shivanshu+Purohit%2C+Lari) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Gao+Leo%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2E+If+you+use+this+software%2C+please+cite+it+using+these)
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
8. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021. Tweepfake: About detecting deepfake tweets. Plos one, 16(5):e0251415. [find](https://scholar.google.com/scholar?q=Tiziano+Fagni%2C+Fabrizio+Falchi%2C+Margherita+Gambini%2C+Antonio+Martella%2C+and+Maurizio+Tesconi%2E+2021%2E+Tweepfake%3A+About+detecting+deepfake+tweets%2E+Plos+one%2C+16%285%29%3Ae0251415%2E)
9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2E)
10. Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694. [find](https://scholar.google.com/scholar?q=Luciano+Floridi+and+Massimo+Chiriatti%2E+2020%2E+Gpt%2D3%3A+Its+nature%2C+scope%2C+limits%2C+and+consequences%2E+Minds+and+Machines%2C+30%3A681%E2%80%93694%2E)
11. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
12. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Hendrik+Strobelt%2C+and+Alexander+M+Rush%2E+2019%2E+Gltr%3A+Statistical+detection+and+visualization+of+generated+text%2E+arXiv+preprint+arXiv%3A1906%2E04043%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/)
13. Alexei Grinbaum and Laurynas Adomaitis. 2022. The ethical need for watermarks in machine-generated language. arXiv preprint arXiv:2209.03118. — [arXiv:2209.03118](https://arxiv.org/abs/2209.03118) [find](https://scholar.google.com/scholar?q=Alexei+Grinbaum+and+Laurynas+Adomaitis%2E+2022%2E+The+ethical+need+for+watermarks+in+machine%2Dgenerated+language%2E+arXiv+preprint+arXiv%3A2209%2E03118%2E+%E2%80%94+%5BarXiv%3A2209%2E03118%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2209%2E03118%29)
14. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. [find](https://scholar.google.com/scholar?q=Biyang+Guo%2C+Xin+Zhang%2C+Ziyuan+Wang%2C+Minqi+Jiang%2C+Jinran+Nie%2C+Yuxuan+Ding%2C+Jianwei+Yue%2C+and+Yupeng+Wu%2E+2023%2E+How+close+is+chatgpt+to+human+experts%3F+comparison+corpus%2C+evaluation%2C+and+detection%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/guo-2023-close-chatgpt-human-experts-comparison/)
15. Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822. — [arXiv:2303.14822](https://arxiv.org/abs/2303.14822) [find](https://scholar.google.com/scholar?q=Xinlei+He%2C+Xinyue+Shen%2C+Zeyuan+Chen%2C+Michael+Backes%2C+and+Yang+Zhang%2E+2023%2E+Mgtbench%3A+Benchmarking+machine%2Dgenerated+text+detection%2E+arXiv+preprint+arXiv%3A2303%2E14822%2E+%E2%80%94+%5BarXiv%3A2303%2E14822%5D%28https%3A%2F%2Farxiv%2E)
16. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2E)
17. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2019. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650. [find](https://scholar.google.com/scholar?q=Daphne+Ippolito%2C+Daniel+Duckworth%2C+Chris+Callison%2DBurch%2C+and+Douglas+Eck%2E+2019%2E+Automatic+detection+of+generated+text+is+easiest+when+humans+are+fooled%2E+arXiv+preprint+arXiv%3A1911%2E00650%2E) [our card](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/)
18. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. 2020. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Muhammad+Abdul%2DMageed%2C+and+Laks+VS+Lakshmanan%2E+2020%2E+Automatic+detection+of+machine+generated+text%3A+A+critical+survey%2E+arXiv+preprint+arXiv%3A2011%2E01314%2E) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/)
19. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. arXiv preprint arXiv:2301.10226. [find](https://scholar.google.com/scholar?q=John+Kirchenbauer%2C+Jonas+Geiping%2C+Yuxin+Wen%2C+Jonathan+Katz%2C+Ian+Miers%2C+and+Tom+Goldstein%2E+2023%2E+A+watermark+for+large+language+models%2E+arXiv+preprint+arXiv%3A2301%2E10226%2E) [our card](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/)
20. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint arXiv:2303.13408. [find](https://scholar.google.com/scholar?q=Kalpesh+Krishna%2C+Yixiao+Song%2C+Marzena+Karpinska%2C+John+Wieting%2C+and+Mohit+Iyyer%2E+2023%2E+Paraphrasing+evades+detectors+of+ai%2Dgenerated+text%2C+but+retrieval+is+an+effective+defense%2E+arXiv+preprint+arXiv%3A23) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2023-paraphrasing-evades-detectors-ai-generated/)
21. Thomas Lavergne, Tanguy Urvoy, and François Yvon. 2008. Detecting fake content with relative entropy scoring. PAN, 8:27–31. [find](https://scholar.google.com/scholar?q=Thomas+Lavergne%2C+Tanguy+Urvoy%2C+and+Fran%C3%A7ois+Yvon%2E+2008%2E+Detecting+fake+content+with+relative+entropy+scoring%2E+PAN%2C+8%3A27%E2%80%9331%2E)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
23. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Yoonho+Lee%2C+Alexander+Khazatsky%2C+Christopher+D+Manning%2C+and+Chelsea+Finn%2E+2023%2E+Detectgpt%3A+Zero%2Dshot+machine%2Dgenerated+text+detection+using+probability+curvature%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/)
24. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+arXiv+preprint+arXiv%3A1808%2E0874)
25. OpenAI. 2022. gpt: Optimizing language models for dialogue. [http://web.archive.org/web/20230109000707/https://openai.com/](http://web.archive.org/web/20230109000707/https://openai.com/) blog/chatgpt/. [↗](http://web.archive.org/web/20230109000707/https://openai.com/)
26. Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. 2022. Threat scenarios and best practices to detect neural fake news. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1233–1249. [find](https://scholar.google.com/scholar?q=Artidoro+Pagnoni%2C+Martin+Graciarena%2C+and+Yulia+Tsvetkov%2E+2022%2E+Threat+scenarios+and+best+practices+to+detect+neural+fake+news%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Ling)
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
29. Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11, page 269. NIH Public Access. [find](https://scholar.google.com/scholar?q=Alexander+Ratner%2C+Stephen+H+Bach%2C+Henry+Ehrenberg%2C+Jason+Fries%2C+Sen+Wu%2C+and+Christopher+R%C3%A9%2E+2017%2E+Snorkel%3A+Rapid+training+data+creation+with+weak+supervision%2E+In+Proceedings+of+the+VLDB+Endowment%2E+Int)
30. Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022. Leveraging large language models for multiple choice question answering. arXiv preprint arXiv:2210.12353. [find](https://scholar.google.com/scholar?q=Joshua+Robinson%2C+Christopher+Michael+Rytting%2C+and+David+Wingate%2E+2022%2E+Leveraging+large+language+models+for+multiple+choice+question+answering%2E+arXiv+preprint+arXiv%3A2210%2E12353%2E) [our card](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/)
31. Kalhan Rosenblatt. 2023. Chatgpt passes mba exam given by a wharton professor. Retrieved Jan, 25:2023. [find](https://scholar.google.com/scholar?q=Kalhan+Rosenblatt%2E+2023%2E+Chatgpt+passes+mba+exam+given+by+a+wharton+professor%2E+Retrieved+Jan%2C+25%3A2023%2E)
32. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156. [find](https://scholar.google.com/scholar?q=Vinu+Sankar+Sadasivan%2C+Aounon+Kumar%2C+Sriram+Balasubramanian%2C+Wenxiao+Wang%2C+and+Soheil+Feizi%2E+2023%2E+Can+ai%2Dgenerated+text+be+reliably+detected%3F+arXiv+preprint+arXiv%3A2303%2E11156%2E) [our card](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/)
33. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
34. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. [find](https://scholar.google.com/scholar?q=Irene+Solaiman%2C+Miles+Brundage%2C+Jack+Clark%2C+Amanda+Askell%2C+Ariel+Herbert%2DVoss%2C+Jeff+Wu%2C+Alec+Radford%2C+Gretchen+Krueger%2C+Jong+Wook+Kim%2C+Sarah+Kreps%2C+et+al%2E+2019%2E+Release+strategies+and+the+social+impac) [our card](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/)
35. Chris Stokel-Walker. 2022. Ai bot chatgpt writes smart essays-should academics worry? Nature. [find](https://scholar.google.com/scholar?q=Chris+Stokel%2DWalker%2E+2022%2E+Ai+bot+chatgpt+writes+smart+essays%2Dshould+academics+worry%3F+Nature%2E)
36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+f)
37. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384–8395. [find](https://scholar.google.com/scholar?q=Adaku+Uchendu%2C+Thai+Le%2C+Kai+Shu%2C+and+Dongwon+Lee%2E+2020%2E+Authorship+attribution+for+neural+text+generation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EM) [our card](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/)
38. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax). [↗](https://github.com/kingoflolz/mesh-transformer-jax)
39. Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces, pages 841–852. [find](https://scholar.google.com/scholar?q=Ann+Yuan%2C+Andy+Coenen%2C+Emily+Reif%2C+and+Daphne+Ippolito%2E+2022%2E+Wordcraft%3A+story+writing+with+large+language+models%2E+In+27th+International+Conference+on+Intelligent+User+Interfaces%2C+pages+841%E2%80%93852%2E)
40. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai… [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Kurt+Shuster%2C+Daniel+S)
41. Xuandong Zhao, Yu-Xiang Wang, and Lei Li. 2023. Protecting language generation models via invisible watermarking. arXiv preprint arXiv:2302.03162. — [arXiv:2302.03162](https://arxiv.org/abs/2302.03162) [find](https://scholar.google.com/scholar?q=Xuandong+Zhao%2C+Yu%2DXiang+Wang%2C+and+Lei+Li%2E+2023%2E+Protecting+language+generation+models+via+invisible+watermarking%2E+arXiv+preprint+arXiv%3A2302%2E03162%2E+%E2%80%94+%5BarXiv%3A2302%2E03162%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2302%2E03162)