---
title: "Mapping the Increasing Use of LLMs in Scientific Papers (2024)"
description: "To build realistic LLM reference text, they had an LLM condense human-written paragraphs into… 79 references, cited by 44 papers here."
authors: ["Weixin Liang", "Yaohui Zhang", "Zhengxuan Wu", "Haley Lepp", "Wenlong Ji", "Xuandong Zhao", "Hancheng Cao", "Sheng Liu", "Siyu He", "Zhi Huang", "Diyi Yang", "Christopher Potts", "Christopher D Manning", "James Y. Zou"]
year: 2024
venue: "arXiv:2404.01268"
source: "https://arxiv.org/abs/2404.01268"
doi: "10.48550/arxiv.2404.01268"
arxiv: "2404.01268"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liang-2024-mapping-increasing-use-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liang-2024-mapping-increasing-use-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Mapping the Increasing Use of LLMs in Scientific Papers

# Mapping the Increasing Use of LLMs in Scientific Papers

**Authors:** [Weixin Liang](https://md.research.elamur.ai/free/en/authors/weixin-liang/), [Yaohui Zhang](https://md.research.elamur.ai/free/en/authors/yaohui-zhang/), [Zhengxuan Wu](https://md.research.elamur.ai/free/en/authors/zhengxuan-wu/), [Haley Lepp](https://md.research.elamur.ai/free/en/authors/haley-lepp/), Wenlong Ji, [Xuandong Zhao](https://md.research.elamur.ai/free/en/authors/xuandong-zhao/), [Hancheng Cao](https://md.research.elamur.ai/free/en/authors/hancheng-cao/), [Sheng Liu](https://md.research.elamur.ai/free/en/authors/sheng-liu/), [Siyu He](https://md.research.elamur.ai/free/en/authors/siyu-he/), [Zhi Huang](https://md.research.elamur.ai/free/en/authors/zhi-huang/), [Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-e-potts/), [Christopher D Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) and [James Y. Zou](https://md.research.elamur.ai/free/en/authors/james-zou/) · **Year:** 2024 · **Venue:** arXiv:2404.01268 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.01268) · [PDF](https://arxiv.org/pdf/2404.01268) · [DOI 10.48550/arxiv.2404.01268](https://doi.org/10.48550/arxiv.2404.01268) · [arXiv 2404.01268](https://arxiv.org/abs/2404.01268) · [OpenAlex](https://openalex.org/W4393905506)

**Citations:** 45 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/liang-2024-mapping-increasing-use-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Since ChatGPT appeared, many have guessed how often researchers use LLMs to write papers, but there has been no reliable measurement. Detectors that classify single documents are unreliable and can be biased against non-native writers. The authors take a different approach. They extend the distributional GPT quantification framework of Liang et al. (2024), which estimates the share of LLM-modified sentences across a whole corpus without labelling any individual text. It does this by fitting a mixture of word-occurrence distributions from human and LLM text using maximum likelihood. To build realistic LLM reference text, they had an LLM condense human-written paragraphs into outlines and then rewrite full paragraphs from those outlines. They applied the method to abstracts and introductions from 950,965 papers published from 2020 to early 2024. LLM-modified content rose sharply about five months after ChatGPT's release, with Computer Science growing fastest. The study offers a scalable way to monitor AI influence on scholarly writing. It also points to possible pressures behind that influence, such as competition and the push to publish quickly.

## Key points
- The corpus covers January 2020 to February 2024. It includes 773,147 arXiv papers (CS, EESS, Mathematics, Physics, Statistics), 161,280 bioRxiv papers and 16,538 articles from 15 Nature portfolio journals.
- By February 2024, the estimated fraction (α) of LLM-modified sentences in arXiv Computer Science reached 17.5% in abstracts and 15.3% in introductions. Mathematics stayed lowest at 4.9% and 3.5%, and the Nature portfolio reached about 6.3% and 6.4%.
- Words that LLMs favour, such as "realm", "intricate", "showcasing" and "pivotal", were rare in arXiv CS abstracts for over a decade. Their use jumped abruptly in 2023.
- Higher α correlates with three features:
  - first authors who post preprints frequently,
  - papers in crowded areas, measured by embedding distance to the closest neighbouring paper,
  - shorter papers. CS papers under about 5,000 words reached 17.7% versus 13.6% for longer ones.
- Estimates for 2022 stayed near zero, suggesting few false positives. The analysis targets ChatGPT only, and the reported associations are correlational rather than causal.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations

📚 Shelves (5): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 52 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 65 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 62 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning and James Y. Zou. 2024. Mapping the Increasing Use of LLMs in Scientific Papers. arXiv:2404.01268.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liang2024mapping,
  title = {{Mapping the Increasing Use of LLMs in Scientific Papers}},
  author = {Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning and James Y. Zou},
  year = {2024},
  doi = {10.48550/arxiv.2404.01268},
  eprint = {2404.01268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.01268}
}
```

- [← previous LVBench: An Extreme Long Video Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-lvbench-extreme-long-video-understanding/)
- [next → BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://md.research.elamur.ai/free/en/papers/kuratov-2024-babilong-testing-limits-llms-long/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

79 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Scott Aaronson. Simons Institute Talk on Watermarking of Large Language Models, 2023. URL [https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17](https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17). [↗](https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17)
2. Mikhail J. Atallah, Victor Raskin, Michael Crogan, Christian F. Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural Language Watermarking: Design, Analysis, and a Proof-of-Concept Implementation. In Information Hiding, 2001. [find](https://scholar.google.com/scholar?q=Mikhail+J%2E+Atallah%2C+Victor+Raskin%2C+Michael+Crogan%2C+Christian+F%2E+Hempelmann%2C+Florian+Kerschbaum%2C+Dina+Mohamed%2C+and+Sanket+Naik%2E+Natural+Language+Watermarking%3A+Design%2C+Analysis%2C+and+a+Proof%2Dof%2DConcept+I)
3. Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. Identifying Real or Fake Articles: Towards better Language Modeling. In International Joint Conference on Natural Language Processing, 2008. [find](https://scholar.google.com/scholar?q=Sameer+Badaskar%2C+Sachin+Agarwal%2C+and+Shilpa+Arora%2E+Identifying+Real+or+Fake+Articles%3A+Towards+better+Language+Modeling%2E+In+International+Joint+Conference+on+Natural+Language+Processing%2C+2008%2E)
4. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. Real or Fake? Learning to Discriminate Machine from Human Generated Text. ArXiv, abs/1906.03351, 2019. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) [find](https://scholar.google.com/scholar?q=Anton+Bakhtin%2C+Sam+Gross%2C+Myle+Ott%2C+Yuntian+Deng%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Arthur+Szlam%2E+Real+or+Fake%3F+Learning+to+Discriminate+Machine+from+Human+Generated+Text%2E+ArXiv%2C+abs%2F1906%2E03351%2C+2019%2E+%E2%80%94+%5BarXi)
5. Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://md.research.elamur.ai/free/en/papers/bao-2023-fast-detectgpt-efficient-zero-shot/). ArXiv, abs/2310.05130, 2023.
6. Daria Beresneva. Computer-Generated Text Detection Using Machine Learning: A Systematic Review. In International Conference on Applications of Natural Language to Data Bases, 2016. [find](https://scholar.google.com/scholar?q=Daria+Beresneva%2E+Computer%2DGenerated+Text+Detection+Using+Machine+Learning%3A+A+Systematic+Review%2E+In+International+Conference+on+Applications+of+Natural+Language+to+Data+Bases%2C+2016%2E)
7. Rahul Bhagat and Eduard H. Hovy. [Squibs: What Is a Paraphrase?](https://md.research.elamur.ai/free/en/papers/bhagat-2013-squibs-what-is-paraphrase/) Computational Linguistics, 39:463–472, 2013.
8. Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. ConDA: Contrastive Domain Adaptation for AI-generated Text Detection. ArXiv, abs/2309.03992, 2023. URL [https://api.semanticscholar.org/CorpusID:261660497](https://api.semanticscholar.org/CorpusID:261660497). [↗](https://api.semanticscholar.org/CorpusID:261660497)
9. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736, 2023. — [arXiv:2304.04736](https://arxiv.org/abs/2304.04736) [find](https://scholar.google.com/scholar?q=Souradip+Chakraborty%2C+Amrit+Singh+Bedi%2C+Sicheng+Zhu%2C+Bang+An%2C+Dinesh+Manocha%2C+and+Furong+Huang%2E+On+the+possibilities+of+ai%2Dgenerated+text+detection%2E+arXiv+preprint+arXiv%3A2304%2E04736%2C+2023%2E+%E2%80%94+%5BarXiv%3A230)
10. Yutian Chen, Hao Kang, Vivian Zhai, Liang Li, Rita Singh, and Bhiksha Ramakrishnan. GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content. ArXiv, abs/2305.07969, 2023. URL [https://api.semanticscholar.org/CorpusID:258686680](https://api.semanticscholar.org/CorpusID:258686680). — [arXiv:2305.07969](https://arxiv.org/abs/2305.07969) [↗](https://api.semanticscholar.org/CorpusID:258686680)
11. Yuei-Lin Chiang, Lu-Ping Chang, Wen-Tai Hsieh, and Wen-Chih Chen. Natural Language Watermarking Using Semantic Substitution for Chinese Text. In International Workshop on Digital Watermarking, 2003. [find](https://scholar.google.com/scholar?q=Yuei%2DLin+Chiang%2C+Lu%2DPing+Chang%2C+Wen%2DTai+Hsieh%2C+and+Wen%2DChih+Chen%2E+Natural+Language+Watermarking+Using+Semantic+Substitution+for+Chinese+Text%2E+In+International+Workshop+on+Digital+Watermarking%2C+2003%2E)
12. Gemma Conroy. How ChatGPT and other AI tools could disrupt scientific publishing. Nature, October 2023a. URL [https://www.nature.com/articles/d41586-023-03144-w](https://www.nature.com/articles/d41586-023-03144-w). [↗](https://www.nature.com/articles/d41586-023-03144-w)
13. Gemma Conroy. Scientific sleuths spot dishonest ChatGPT use in papers. Nature, September 2023b. URL [https://www.nature.com/articles/d41586-023-02477-w](https://www.nature.com/articles/d41586-023-02477-w). [↗](https://www.nature.com/articles/d41586-023-02477-w)
14. Evan Crothers, Nathalie Japkowicz, and Herna Viktor. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. arXiv preprint arXiv:2210.07321, 2022. [find](https://scholar.google.com/scholar?q=Evan+Crothers%2C+Nathalie+Japkowicz%2C+and+Herna+Viktor%2E+Machine+Generated+Text%3A+A+Comprehensive+Survey+of+Threat+Models+and+Detection+Methods%2E+arXiv+preprint+arXiv%3A2210%2E07321%2C+2022%2E)
15. Mack Deguerin. AI-generated nonsense is leaking into scientific journals . Popular Science, March 2024. URL [https://www.popsci.com/technology/ai-generated-text-scientific-journals/](https://www.popsci.com/technology/ai-generated-text-scientific-journals/). [↗](https://www.popsci.com/technology/ai-generated-text-scientific-journals/)
16. Fabrizio Dell’Acqua, Edward McFowland, Ethan R Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R Lakhani. Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and q… [find](https://scholar.google.com/scholar?q=Fabrizio+Dell%E2%80%99Acqua%2C+Edward+McFowland%2C+Ethan+R+Mollick%2C+Hila+Lifshitz%2DAssaf%2C+Katherine+Kellogg%2C+Saran+Rajendran%2C+Lisa+Krayer%2C+Franc%CC%A7ois+Candelon%2C+and+Karim+R+Lakhani%2E+Navigating+the+jagged+technologic)
17. Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What’s In My Big Data? In The Twelfth International Conference on Learning Representations, 2023. — [What's In My Big Data?](https://md.research.elamur.ai/free/en/papers/elazar-2023-s-my-big-data/)
18. Holly Else. Abstracts written by ChatGPT fool scientists. Nature, Jan 2023. URL [https://www.nature.com/articles/d41586-023-00056-7](https://www.nature.com/articles/d41586-023-00056-7). [↗](https://www.nature.com/articles/d41586-023-00056-7)
19. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. [TweepFake: About detecting deepfake tweets](https://md.research.elamur.ai/free/en/papers/fagni-2020-tweepfake-about-detecting-deepfake-tweets/). Plos one, 16(5):e0251415, 2021.
20. Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three Bricks to Consolidate Watermarks for Large Language Models. 2023 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6, 2023. [find](https://scholar.google.com/scholar?q=Pierre+Fernandez%2C+Antoine+Chaffin%2C+Karim+Tit%2C+Vivien+Chappelier%2C+and+Teddy+Furon%2E+Three+Bricks+to+Consolidate+Watermarks+for+Large+Language+Models%2E+2023+IEEE+International+Workshop+on+Information+Fore)
21. Jacob G Foster, Andrey Rzhetsky, and James A Evans. Tradition and innovation in scientists’ research strategies. American sociological review, 80(5):875–908, 2015. [find](https://scholar.google.com/scholar?q=Jacob+G+Foster%2C+Andrey+Rzhetsky%2C+and+James+A+Evans%2E+Tradition+and+innovation+in+scientists%E2%80%99+research+strategies%2E+American+sociological+review%2C+80%285%29%3A875%E2%80%93908%2C+2015%2E)
22. Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. Comparing scientific abstracts generated by Chat-GPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv… [find](https://scholar.google.com/scholar?q=Catherine+A+Gao%2C+Frederick+M+Howard%2C+Nikolay+S+Markov%2C+Emma+C+Dyer%2C+Siddhi+Ramesh%2C+Yuan+Luo%2C+and+Alexander+T+Pearson%2E+Comparing+scientific+abstracts+generated+by+Chat%2DGPT+to+original+abstracts+using+a)
23. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
24. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. [GLTR: Statistical Detection and Visualization of Generated Text](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 111–116, 2019.
25. Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and A. S. Bedi. Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey. ArXiv, abs/2310.15264, 2023. — [arXiv:2310.15264](https://arxiv.org/abs/2310.15264) [find](https://scholar.google.com/scholar?q=Soumya+Suvra+Ghosal%2C+Souradip+Chakraborty%2C+Jonas+Geiping%2C+Furong+Huang%2C+Dinesh+Manocha%2C+and+A%2E+S%2E+Bedi%2E+Towards+Possibilities+%26+Impossibilities+of+AI%2Dgenerated+Text+Detection%3A+A+Survey%2E+ArXiv%2C+abs%2F231)
26. Sourojit Ghosh and Aylin Caliskan. ’Person’== Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion. arXiv preprint arXiv:2310.19981, 2023. [find](https://scholar.google.com/scholar?q=Sourojit+Ghosh+and+Aylin+Caliskan%2E+%E2%80%99Person%E2%80%99%3D%3D+Light%2Dskinned%2C+Western+Man%2C+and+Sexualization+of+Women+of+Color%3A+Stereotypes+in+Stable+Diffusion%2E+arXiv+preprint+arXiv%3A2310%2E19981%2C+2023%2E)
27. Melissa Heikkilä. How to spot AI-generated text. MIT Technology Review, Dec 2022. URL [https://www.technologyreview.com/2022/12/19/1065596/how-to-spot-ai-generated-text/](https://www.technologyreview.com/2022/12/19/1065596/how-to-spot-ai-generated-text/). [↗](https://www.technologyreview.com/2022/12/19/1065596/how-to-spot-ai-generated-text/)
28. Xiaobing Hu, Pin-Yu Chen, and Tsung-Yi Ho. [RADAR: Robust AI-Text Detection via Adversarial Learning](https://md.research.elamur.ai/free/en/papers/hu-2023-radar-robust-ai-text-detection/). ArXiv, abs/2307.03838, 2023a. URL [https://api.semanticscholar.org/CorpusID:259501842](https://api.semanticscholar.org/CorpusID:259501842). [↗](https://api.semanticscholar.org/CorpusID:259501842)
29. Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. Unbiased Watermark for Large Language Models. ArXiv, abs/2310.10669, 2023b. — [arXiv:2310.10669](https://arxiv.org/abs/2310.10669) [find](https://scholar.google.com/scholar?q=Zhengmian+Hu%2C+Lichang+Chen%2C+Xidong+Wu%2C+Yihan+Wu%2C+Hongyang+Zhang%2C+and+Heng+Huang%2E+Unbiased+Watermark+for+Large+Language+Models%2E+ArXiv%2C+abs%2F2310%2E10669%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2310%2E10669%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
30. ICML. Clarification on large language model policy LLM. [https://icml.cc/Conferences/2023/llm-policy](https://icml.cc/Conferences/2023/llm-policy), 2023. [↗](https://icml.cc/Conferences/2023/llm-policy)
31. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. [Automatic detection of generated text is easiest when humans are fooled](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/). arXiv preprint arXiv:1911.00650, 2019.
32. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. [Automatic detection of machine generated text: A critical survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/). arXiv preprint arXiv:2011.01314, 2020.
33. Samantha Murphy Kelly. ChatGPT creator pulls AI detection tool due to ‘low rate of accuracy’. CNN Business, Jul 2023. URL [https://www.cnn.com/2023/07/25/tech/openai-ai-detection-tool/index.html](https://www.cnn.com/2023/07/25/tech/openai-ai-detection-tool/index.html). [↗](https://www.cnn.com/2023/07/25/tech/openai-ai-detection-tool/index.html)
34. Saurabh Khanna, Jon Ball, Juan Pablo Alperin, and John Willinsky. Recalibrating the scope of scholarly publishing: A modest step in a vast decolonization process. Quantitative Science Studies, 3(4):912–930, 12 2022. ISSN 2641-3337. doi: 10.1162/qss a 00228. URL https://doi.org/10.1162/qss a 00228. — [doi:10.1162/qss](https://doi.org/10.1162/qss) [↗](https://doi.org/10.1162/qss)
35. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). International Conference on Machine Learning, 2023.
36. Jan Hendrik Kirchner, Lama Ahmad, Scott Aaronson, and Jan Leike. New AI classifier for indicating AI-written text, 2023. URL [https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text](https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text). [↗](https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text)
37. Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. [Robust Distortion-free Watermarks for Language Models](https://md.research.elamur.ai/free/en/papers/kuditipudi-2023-robust-distortion-free-watermarks-language/). ArXiv, abs/2307.15593, 2023.
38. Thomas Lavergne, Tanguy Urvoy, and François Yvon. Detecting Fake Content with Relative Entropy Scoring. Pan, 2008. [find](https://scholar.google.com/scholar?q=Thomas+Lavergne%2C+Tanguy+Urvoy%2C+and+Franc%CC%A7ois+Yvon%2E+Detecting+Fake+Content+with+Relative+Entropy+Scoring%2E+Pan%2C+2008%2E)
39. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Deepfake Text Detection in the Wild. ArXiv, abs/2305.13242, 2023. URL [https://api.semanticscholar.org/CorpusID:258832454](https://api.semanticscholar.org/CorpusID:258832454). — [arXiv:2305.13242](https://arxiv.org/abs/2305.13242) [↗](https://api.semanticscholar.org/CorpusID:258832454)
40. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Y. Zou. [GPT detectors are biased against non-native English writers](https://md.research.elamur.ai/free/en/papers/liang-2023-gpt-detectors-biased-against-non/). ArXiv, abs/2304.02819, 2023a.
41. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, et al. [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://md.research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/). arXiv preprint arXiv:2310.01783, 2023b.
42. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://md.research.elamur.ai/free/en/papers/liang-2024-monitoring-ai-modified-content/). arXiv preprint arXi…
43. Ryan Liu and Nihar B Shah. [Reviewergpt? an exploratory study on using large language models for paper reviewing](https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/). arXiv preprint arXiv:2306.00622, 2023.
44. Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, and Chao Shen. CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning. ArXiv, abs/2212.10341, 2022. URL [https://api.semanticscholar.org/CorpusID:254877728](https://api.semanticscholar.org/CorpusID:254877728). — [arXiv:2212.10341](https://arxiv.org/abs/2212.10341) [↗](https://api.semanticscholar.org/CorpusID:254877728)
45. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). ArXiv, abs/1907.11692, 2019.
46. MacroPolo. The Global AI Talent Tracker, 2024. digital-projects/the-global-ai-talent-tracker/. [find](https://scholar.google.com/scholar?q=MacroPolo%2E+The+Global+AI+Talent+Tracker%2C+2024%2E+digital%2Dprojects%2Fthe%2Dglobal%2Dai%2Dtalent%2Dtracker%2F%2E)
47. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/). ArXiv, abs/2301.11305, 2023a.
48. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. [DetectGPT: Zero-shot machine-generated text detection using probability curvature](https://md.research.elamur.ai/free/en/papers/mitchell-2023-detectgpt-zero-shot-machine-generated/). arXiv preprint arXiv:2301.11305, 2023b.
49. Paulina Okunytė. Google search exposes academics using ChatGPT in research papers. Cybernews, November 2023. URL [https://cybernews.com/news/academic-cheating-chatgpt-openai/](https://cybernews.com/news/academic-cheating-chatgpt-openai/). [↗](https://cybernews.com/news/academic-cheating-chatgpt-openai/)
50. URL [https://macropolo.org/OpenAI](https://macropolo.org/OpenAI). GPT-2: 1.5B release. https://openai.com/research/gpt-2-1-5b-release, 2019. Accessed: 2019-11-05. [↗](https://macropolo.org/OpenAI)
51. Ivan Oransky and Adam Marcus. Papers and peer reviews with evidence of ChatGPT writing . Retraction Watch, 2024. URL [https://retractionwatch.com/papers-and-peer-reviews-with-evidence-of-chatgpt-writing/](https://retractionwatch.com/papers-and-peer-reviews-with-evidence-of-chatgpt-writing/). [↗](https://retractionwatch.com/papers-and-peer-reviews-with-evidence-of-chatgpt-writing/)
52. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
53. Vinu Sankar Sadasivan, Aounon Kumar, S. Balasubramanian, Wenxiao Wang, and Soheil Feizi. [Can AI-Generated Text be Reliably Detected?](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/) ArXiv, abs/2303.11156, 2023.
54. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In International Conference on Machine Learning, pp. 29971–30004. PMLR, 2023.
55. Zhouxing Shi, Yihan Wang, Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsieh. [Red Teaming Language Model Detectors with Language Models](https://md.research.elamur.ai/free/en/papers/shi-2024-red-teaming-language-model-detectors/). ArXiv, abs/2305.19713, 2023.
56. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/). arXiv preprint arXiv:2305.17493, 2023.
57. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. [Release strategies and the social impacts of language models](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/). arXiv preprint arXiv:1908.09203, 2019.
58. H. Holden Thorp. Chatgpt is fun, but not an author. Science, 379(6630):313–313, 2023. doi: 10.1126/science.adg7879. URL [https://www.science.org/doi/abs/10.1126/science](https://www.science.org/doi/abs/10.1126/science). adg7879. — [doi:10.1126/science.adg7879](https://doi.org/10.1126/science.adg7879) [↗](https://www.science.org/doi/abs/10.1126/science)
59. Mercan Topkara, Giuseppe Riccardi, Dilek Z. Hakkani-Tür, and Mikhail J. Atallah. Natural language watermarking: challenges in building a practical system. In Electronic imaging, 2006a. [find](https://scholar.google.com/scholar?q=Mercan+Topkara%2C+Giuseppe+Riccardi%2C+Dilek+Z%2E+Hakkani%2DTu%CC%88r%2C+and+Mikhail+J%2E+Atallah%2E+Natural+language+watermarking%3A+challenges+in+building+a+practical+system%2E+In+Electronic+imaging%2C+2006a%2E)
60. Umut Topkara, Mercan Topkara, and Mikhail J. Atallah. The hiding virtues of ambiguity: quantifiably resilient watermarking of natural language text through synonym substitutions. In Workshop on Multimedia & Security, 2006b. [find](https://scholar.google.com/scholar?q=Umut+Topkara%2C+Mercan+Topkara%2C+and+Mikhail+J%2E+Atallah%2E+The+hiding+virtues+of+ambiguity%3A+quantifiably+resilient+watermarking+of+natural+language+text+through+synonym+substitutions%2E+In+Workshop+on+Multim)
61. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
62. Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, S. Barannikov, Irina Piontkovskaya, Sergey I. Nikolenko, and Evgeny Burnaev. [Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts](https://md.research.elamur.ai/free/en/papers/tulchinskii-2023-intrinsic-dimension-estimation-robust-detection/). ArXiv, abs/2306.04723, 2023.
63. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. [Authorship Attribution for Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/). In Conference on Empirical Methods in Natural Language Processing, 2020.
64. Dann. Van Rossum. Generative AI Top 150: The World’s Most Used AI Tools. [https://www.flexos.work/learn/generative-ai-top-150](https://www.flexos.work/learn/generative-ai-top-150), February 2024. [↗](https://www.flexos.work/learn/generative-ai-top-150)
65. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/). arXiv preprint arXiv:2306.07899, 2023.
66. James Vincent. ‘As an AI language model’: the phrase that shows how AI is pollulating the web. The Verge, Apr 2023. URL [https://www.theverge.com/2023/4/25/23697218/ai-generated-spam-fake-user-reviews-as-an-ai-language-model](https://www.theverge.com/2023/4/25/23697218/ai-generated-spam-fake-user-reviews-as-an-ai-language-model). [↗](https://www.theverge.com/2023/4/25/23697218/ai-generated-spam-fake-user-reviews-as-an-ai-language-model)
67. Debora Weber-Wulff, Alla Anohina-Naumeca, Sonja Bjelobaba, Tomáš Foltýnek, Jean Guerrero-Dib, Olumide Popoola, Petr Šigut, and Lorna Waddington. Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1):26, 2023. ISSN 1833-2595. doi: 10.1007/s40979-… — [doi:10.1007/s40979-023-00146-z](https://doi.org/10.1007/s40979-023-00146-z) [find](https://scholar.google.com/scholar?q=Debora+Weber%2DWulff%2C+Alla+Anohina%2DNaumeca%2C+Sonja+Bjelobaba%2C+Toma%CC%81s%CC%8C+Folty%CC%81nek%2C+Jean+Guerrero%2DDib%2C+Olumide+Popoola%2C+Petr+S%CC%8Cigut%2C+and+Lorna+Waddington%2E+Testing+of+detection+tools+for+AI%2Dgenerated+text%2E+I)
68. Max Wolff. Attacking Neural Text Detectors. ArXiv, abs/2002.11768, 2020. — [arXiv:2002.11768](https://arxiv.org/abs/2002.11768) [find](https://scholar.google.com/scholar?q=Max+Wolff%2E+Attacking+Neural+Text+Detectors%2E+ArXiv%2C+abs%2F2002%2E11768%2C+2020%2E+%E2%80%94+%5BarXiv%3A2002%2E11768%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E11768%29)
69. Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. DiPmark: A Stealthy, Efficient and Resilient Watermark for Large Language Models. ArXiv, abs/2310.07710, 2023. — [arXiv:2310.07710](https://arxiv.org/abs/2310.07710) [find](https://scholar.google.com/scholar?q=Yihan+Wu%2C+Zhengmian+Hu%2C+Hongyang+Zhang%2C+and+Heng+Huang%2E+DiPmark%3A+A+Stealthy%2C+Efficient+and+Resilient+Watermark+for+Large+Language+Models%2E+ArXiv%2C+abs%2F2310%2E07710%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E07710%5D%28https%3A%2F%2Farxi)
70. Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text. ArXiv, abs/2305.17359, 2023a. — [arXiv:2305.17359](https://arxiv.org/abs/2305.17359) [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Wei+Cheng%2C+Linda+Petzold%2C+William+Yang+Wang%2C+and+Haifeng+Chen%2E+DNA%2DGPT%3A+Divergent+N%2DGram+Analysis+for+Training%2DFree+Detection+of+GPT%2DGenerated+Text%2E+ArXiv%2C+abs%2F2305%2E17359%2C+2023a%2E+%E2%80%94+%5BarXi)
71. Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. A Survey on Detection of LLMs-Generated Content. ArXiv, abs/2310.15654, 2023b. [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Liangming+Pan%2C+Xuandong+Zhao%2C+Haifeng+Chen%2C+Linda+Ruth+Petzold%2C+William+Yang+Wang%2C+and+Wei+Cheng%2E+A+Survey+on+Detection+of+LLMs%2DGenerated+Content%2E+ArXiv%2C+abs%2F2310%2E15654%2C+2023b%2E)
72. Kiyoon Yoo, Wonhyuk Ahn, Jiho Jang, and No Jun Kwak. Robust Multi-bit Natural Language Watermarking through Invariant Features. In Annual Meeting of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Kiyoon+Yoo%2C+Wonhyuk+Ahn%2C+Jiho+Jang%2C+and+No+Jun+Kwak%2E+Robust+Multi%2Dbit+Natural+Language+Watermarking+through+Invariant+Features%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2023)
73. Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Neng H. Yu. GPT Paternity Test: GPT Generated Text Detection with GPT Genetic Inheritance. ArXiv, abs/2305.12519, 2023. URL [https://api.semanticscholar.org/CorpusID:258833423](https://api.semanticscholar.org/CorpusID:258833423). — [arXiv:2305.12519](https://arxiv.org/abs/2305.12519) [↗](https://api.semanticscholar.org/CorpusID:258833423)
74. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. [Defending Against Neural Fake News](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/). ArXiv, abs/1905.12616, 2019.
75. Yi-Fan Zhang, Zhang Zhang, Liang Wang, Tien-Ping Tan, and Rong Jin. Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors. ArXiv, abs/2312.12918, 2023. — [arXiv:2312.12918](https://arxiv.org/abs/2312.12918) [find](https://scholar.google.com/scholar?q=Yi%2DFan+Zhang%2C+Zhang+Zhang%2C+Liang+Wang%2C+Tien%2DPing+Tan%2C+and+Rong+Jin%2E+Assaying+on+the+Robustness+of+Zero%2DShot+Machine%2DGenerated+Text+Detectors%2E+ArXiv%2C+abs%2F2312%2E12918%2C+2023%2E+%E2%80%94+%5BarXiv%3A2312%2E12918%5D%28https%3A%2F%2F)
76. Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting Language Generation Models via Invisible Watermarking. In Proceedings of the 40th International Conference on Machine Learning, pp. 42187–42199, 2023. [find](https://scholar.google.com/scholar?q=Xuandong+Zhao%2C+Yu%2DXiang+Wang%2C+and+Lei+Li%2E+Protecting+Language+Generation+Models+via+Invisible+Watermarking%2E+In+Proceedings+of+the+40th+International+Conference+on+Machine+Learning%2C+pp%2E+42187%E2%80%9342199%2C+20)
77. Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. [Provable Robust Watermarking for AI-Generated Text](https://md.research.elamur.ai/free/en/papers/zhao-2023-provable-robust-watermarking-ai-generated/). In International Conference on Learning Representations (ICLR), 2024a.
78. Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Permute-and-Flip: An optimally robust and watermarkable decoder for LLMs. arXiv preprint arXiv:2402.05864, 2024b. — [arXiv:2402.05864](https://arxiv.org/abs/2402.05864) [find](https://scholar.google.com/scholar?q=Xuandong+Zhao%2C+Lei+Li%2C+and+Yu%2DXiang+Wang%2E+Permute%2Dand%2DFlip%3A+An+optimally+robust+and+watermarkable+decoder+for+LLMs%2E+arXiv+preprint+arXiv%3A2402%2E05864%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2402%2E05864%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
79. Estimated Fraction of LLM-Modified Sentences in Introductions [find](https://scholar.google.com/scholar?q=Estimated+Fraction+of+LLM%2DModified+Sentences+in+Introductions)

*Updated 2 November 2026*