---
title: "Speak, Memory — Chang 2023"
description: "Large language models such as ChatGPT and GPT-4 are increasingly used in cultural analytics, but their training data is undisclosed."
authors: ["Kent K. Chang", "Mackenzie Cramer", "Sandeep Soni", "David Bamman"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.453/"
doi: "10.18653/v1/2023.emnlp-main.453"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

# Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

**Authors:** Kent K. Chang, Mackenzie Cramer, Sandeep Soni, [David Bamman](https://md.research.elamur.ai/free/en/authors/david-bamman/) · **Year:** 2023 · **Venue:** Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.453/) · [PDF](https://aclanthology.org/2023.emnlp-main.453.pdf) · [DOI 10.18653/v1/2023.emnlp-main.453](https://doi.org/10.18653/v1/2023.emnlp-main.453) · [OpenAlex](https://openalex.org/W4389519044)

**Citations:** 83 citations (openalex-snapshot · 2026-10-14) · [83](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Large language models such as ChatGPT and GPT-4 are increasingly used in cultural analytics, but their training data is undisclosed. This makes it hard to know whether evaluation texts were already seen during training. The authors run a "data archaeology" to estimate which works of fiction these models know. They use a membership inference probe called name cloze: the model must recover a single masked character name in a short passage that contains no other named entities.

The probe covers 571 English-language novels published between 1749 and 2020, drawn from LitBank, Pulitzer nominees, bestsellers, works by Black authors and other sources. The results show that OpenAI models, GPT-4 in particular, have memorized many in-copyright books. Memorization tracks how often a book's passages appear on the web. Downstream tasks also go better on memorized titles. The authors conclude that closed models undermine measurement validity in literary research, and they advocate using open models with publicly documented training data.

## Key points

- **Evaluation set:** BookNLP extracted passages of 40–60 tokens, each with exactly one person name and no other entities. The authors sampled 100 passages per book. Always guessing the most frequent name ("Mary") scores 0.6%, and a human annotator scored 0%.
- **What is memorized:** The most memorized books are science fiction/fantasy, popular public-domain classics and bestsellers. Science fiction/fantasy accounts for 12 of the top 20 memorized in-copyright titles, including Harry Potter, 1984 and Dune.
- **Link to web frequency:** Memorization levels align with how often a book appears in Google, Bing and C4 results. Web prevalence therefore offers a rough indicator of contamination risk.
- **Effect on downstream tasks:** The models perform better on memorized books when predicting a book's first publication year and estimating narrative time duration.
- **BERT finding:** BookCorpus contains copyrighted novels such as Fifty Shades of Grey and Outlander, and BERT has memorized them. Code and data are available at github.com/bamman-group/gpt4-books.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 78 not yet.

- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/)
- [Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist](https://md.research.elamur.ai/free/en/papers/karpinska-2023-large-language-models-effectively-leverage/)
- [S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature](https://md.research.elamur.ai/free/en/papers/pardo-2026-s-vocal-dataset-evaluation-framework/)
- [EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMs](https://md.research.elamur.ai/free/en/papers/satvaty-2026-el-mia-quantifying-membership-inference/)
- [Memorization or Lucky Guesses: Detecting Short Sequences from Copyrighted Dutch News in LLM Output](https://md.research.elamur.ai/free/en/papers/veerbeek-2026-memorization-or-lucky-guesses-detecting/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chang2023speak,
  title = {{Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4}},
  author = {Chang, Kent K. and Cramer, Mackenzie and Soni, Sandeep and Bamman, David},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.453},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.453}
}
```

- [← previous Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
- [next → Specializing Smaller Language Models towards Multi-Step Reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

66 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Bamman, Michelle Carney, Jon Gillick, Cody Hennesy, and Vijitha Sridhar. 2017. Estimating the date of first publication in a large-scale digital library. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–10. IEEE. [find](https://scholar.google.com/scholar?q=David+Bamman%2C+Michelle+Carney%2C+Jon+Gillick%2C+Cody+Hennesy%2C+and+Vijitha+Sridhar%2E+2017%2E+Estimating+the+date+of+first+publication+in+a+large%2Dscale+digital+library%2E+In+2017+ACM%2FIEEE+Joint+Conference+on+Dig)
2. David M. Berry. 2011. The computational turn: Thinking about the digital humanities. Culture Machine, 12. [find](https://scholar.google.com/scholar?q=David+M%2E+Berry%2E+2011%2E+The+computational+turn%3A+Thinking+about+the+digital+humanities%2E+Culture+Machine%2C+12%2E)
3. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A+A+suite)
4. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2022%2E+Language+contamination+helps+explains+the+cross%2Dlingual+capabilities+of+English+pretrained+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+)
5. Pierre Bourdieu. 1987. Distinction: A social critique of the judgement of taste. Harvard university press. [find](https://scholar.google.com/scholar?q=Pierre+Bourdieu%2E+1987%2E+Distinction%3A+A+social+critique+of+the+judgement+of+taste%2E+Harvard+university+press%2E)
6. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188. — [arXiv:2301.13188](https://arxiv.org/abs/2301.13188) [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Jamie+Hayes%2C+Milad+Nasr%2C+Matthew+Jagielski%2C+Vikash+Sehwag%2C+Florian+Tramer%2C+Borja+Balle%2C+Daphne+Ippolito%2C+and+Eric+Wallace%2E+2023%2E+Extracting+training+data+from+diffusion+models%2E+arXiv)
7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Daphne+Ippolito%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Florian+Tramer%2C+and+Chiyuan+Zhang%2E+2022%2E+Quantifying+memorization+across+neural+language+models%2E+arXiv+preprint+arXiv%3A2202%2E07646%2E)
8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6. [find](https://scholar.google.com/scholar?q=Nicholas+Carlini%2C+Florian+Tramer%2C+Eric+Wallace%2C+Matthew+Jagielski%2C+Ariel+Herbert%2DVoss%2C+Katherine+Lee%2C+Adam+Roberts%2C+Tom+B+Brown%2C+Dawn+Song%2C+Ulfar+Erlingsson%2C+et+al%2E+2021%2E+Extracting+training+data+from) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
9. Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. 2020. Living machines: A study of atypical animacy. In Proceedings of the 28th International Conference on Computational Ling… [find](https://scholar.google.com/scholar?q=Mariona+Coll+Ardanuy%2C+Federico+Nanni%2C+Kaspar+Beelen%2C+Kasra+Hosseini%2C+Ruth+Ahnert%2C+Jon+Lawrence%2C+Katherine+McDonough%2C+Giorgia+Tolfo%2C+Daniel+CS+Wilson%2C+and+Barbara+McGillivray%2E+2020%2E+Living+machines%3A+A+)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
11. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Gili+Baumer%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+)
13. M Elam. 2023. Poetry Will Not Optimize, or What Is Literature to AI? American literature; a journal of literary history, criticism and bibliography. [find](https://scholar.google.com/scholar?q=M+Elam%2E+2023%2E+Poetry+Will+Not+Optimize%2C+or+What+Is+Literature+to+AI%3F+American+literature%3B+a+journal+of+literary+history%2C+criticism+and+bibliography%2E)
14. Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1325–1335, Online. Asso… [find](https://scholar.google.com/scholar?q=Aparna+Elangovan%2C+Jiayuan+He%2C+and+Karin+Verspoor%2E+2021%2E+Memorization+vs%2E+generalization+%3A+Quantifying+data+leakage+in+NLP+performance+evaluation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+)
15. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring causal effects of data statistics on language model’s ‘factual’ predictions. arXiv preprint arXiv:2207.14251. — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Amir+Feder%2C+Abhilasha+Ravichander%2C+Marius+Mosbach%2C+Yonatan+Belinkov%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2022%2E+Measuring+causal+effects+of+data+statistics+)
16. Katherine Elkins and Jon Chun. 2020. Can GPT-3 pass a writer’s Turing test? Journal of cultural analytics, 5(2). [find](https://scholar.google.com/scholar?q=Katherine+Elkins+and+Jon+Chun%2E+2020%2E+Can+GPT%2D3+pass+a+writer%E2%80%99s+Turing+test%3F+Journal+of+cultural+analytics%2C+5%282%29%2E)
17. Elizabeth F. Evans and Matthew Wilkens. 2018. Nation, ethnicity, and the geography of British fiction, 1880– 1940. Cultural Analytics. [find](https://scholar.google.com/scholar?q=Elizabeth+F%2E+Evans+and+Matthew+Wilkens%2E+2018%2E+Nation%2C+ethnicity%2C+and+the+geography+of+British+fiction%2C+1880%E2%80%93+1940%2E+Cultural+Analytics%2E)
18. Kathleen Fitzpatrick. 2012. The humanities, done digitally. Debates in the digital humanities, pages 12–15. [find](https://scholar.google.com/scholar?q=Kathleen+Fitzpatrick%2E+2012%2E+The+humanities%2C+done+digitally%2E+Debates+in+the+digital+humanities%2C+pages+12%E2%80%9315%2E)
19. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
20. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daum%C3%A9+Iii%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Communications+of+the+ACM%2C+64%2812%29%3A86%E2%80%93+92%2E)
21. Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. [find](https://scholar.google.com/scholar?q=Hila+Gonen%2C+Srini+Iyer%2C+Terra+Blevins%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2022%2E+Demystifying+prompts+in+language+models+via+perplexity+estimation%2E)
22. Lauren M E Goodlad and Wai Chee Dimock. 2021. AI and the Human. PMLA, 136(2):317–319. [find](https://scholar.google.com/scholar?q=Lauren+M+E+Goodlad+and+Wai+Chee+Dimock%2E+2021%2E+AI+and+the+Human%2E+PMLA%2C+136%282%29%3A317%E2%80%93319%2E)
23. John Guillory. 1993. Cultural capital the problem of literary canon formation. University of Chicago Press, Chicago. [find](https://scholar.google.com/scholar?q=John+Guillory%2E+1993%2E+Cultural+capital+the+problem+of+literary+canon+formation%2E+University+of+Chicago+Press%2C+Chicago%2E)
24. Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. 2022. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Dallas+Card%2C+Sarah+K+Drier%2C+Emily+K+Gade%2C+Leroy+Z+Wang%2C+Zeyu+Wang%2C+Luke+Zettlemoyer%2C+and+Noah+A+Smith%2E+2022%2E+Whose+language+counts+as+high+quality%3F+Measuring+language+ideologies+in+)
25. Gary Gutting. 1989. Michel Foucault’s Archaeology of Scientific Reason: Science and the History of Reason. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Gary+Gutting%2E+1989%2E+Michel+Foucault%E2%80%99s+Archaeology+of+Scientific+Reason%3A+Science+and+the+History+of+Reason%2E+Cambridge+University+Press%2E)
26. Sil Hamilton and Andrew Piper. 2022. The COVID that wasn’t: Counterfactual journalism using GPT. In Proceedings of the 6th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 83–93, Gyeongju, Republic of Korea. International Con… [find](https://scholar.google.com/scholar?q=Sil+Hamilton+and+Andrew+Piper%2E+2022%2E+The+COVID+that+wasn%E2%80%99t%3A+Counterfactual+journalism+using+GPT%2E+In+Proceedings+of+the+6th+Joint+SIGHUM+Workshop+on+Computational+Linguistics+for+Cultural+Heritage%2C+Soc)
27. N. Katherine Hayles. 2012. How We Think: Transforming Power and Digital Technologies. In David M Berry, editor, Understanding Digital Humanities, pages 42–66. Palgrave Macmillan UK, London. [find](https://scholar.google.com/scholar?q=N%2E+Katherine+Hayles%2E+2012%2E+How+We+Think%3A+Transforming+Power+and+Digital+Technologies%2E+In+David+M+Berry%2C+editor%2C+Understanding+Digital+Humanities%2C+pages+42%E2%80%9366%2E+Palgrave+Macmillan+UK%2C+London%2E)
28. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. 2023. Foundation models and fair use. arXiv preprint arXiv:2303.15715. — [arXiv:2303.15715](https://arxiv.org/abs/2303.15715) [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Xuechen+Li%2C+Dan+Jurafsky%2C+Tatsunori+Hashimoto%2C+Mark+A+Lemley%2C+and+Percy+Liang%2E+2023%2E+Foundation+models+and+fair+use%2E+arXiv+preprint+arXiv%3A2303%2E15715%2E+%E2%80%94+%5BarXiv%3A2303%2E15715%5D%28https%3A%2F%2Farxi)
29. Leah Henrickson and Albert Meroño-Peñuela. 2022. The Hermeneutics of Computer-Generated Texts. Configurations, 30(2):115–139. [find](https://scholar.google.com/scholar?q=Leah+Henrickson+and+Albert+Mero%C3%B1o%2DPe%C3%B1uela%2E+2022%2E+The+Hermeneutics+of+Computer%2DGenerated+Texts%2E+Configurations%2C+30%282%29%3A115%E2%80%93139%2E)
30. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children’s books with explicit memory representations. ICLR. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+The+Goldilocks+principle%3A+Reading+children%E2%80%99s+books+with+explicit+memory+representations%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/)
31. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628. [find](https://scholar.google.com/scholar?q=Jie+Huang%2C+Hanyin+Shao%2C+and+Kevin+Chen%2DChuan+Chang%2E+2022%2E+Are+large+pre%2Dtrained+language+models+leaking+your+personal+information%3F+arXiv+preprint+arXiv%3A2205%2E12628%2E)
32. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) [find](https://scholar.google.com/scholar?q=Daphne+Ippolito%2C+Florian+Tram%C3%A8r%2C+Milad+Nasr%2C+Chiyuan+Zhang%2C+Matthew+Jagielski%2C+Katherine+Lee%2C+Christopher+A+Choquette%2DChoo%2C+and+Nicholas+Carlini%2E+2022%2E+Preventing+verbatim+memorization+in+language+mod)
33. FM Jong, Henning Rode, and Djoerd Hiemstra. 2005. Temporal language models for the disclosure of historical text. Proceedings of the XVIth International Conference of the Association for History and Computing (AHC 2005). [find](https://scholar.google.com/scholar?q=FM+Jong%2C+Henning+Rode%2C+and+Djoerd+Hiemstra%2E+2005%2E+Temporal+language+models+for+the+disclosure+of+historical+text%2E+Proceedings+of+the+XVIth+International+Conference+of+the+Association+for+History+and+C)
34. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022a. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411. [find](https://scholar.google.com/scholar?q=Nikhil+Kandpal%2C+Haikang+Deng%2C+Adam+Roberts%2C+Eric+Wallace%2C+and+Colin+Raffel%2E+2022a%2E+Large+language+models+struggle+to+learn+long%2Dtail+knowledge%2E+arXiv+preprint+arXiv%3A2211%2E08411%2E) [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/)
35. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022b. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. PMLR. [find](https://scholar.google.com/scholar?q=Nikhil+Kandpal%2C+Eric+Wallace%2C+and+Colin+Raffel%2E+2022b%2E+Deduplicating+training+data+mitigates+privacy+risks+in+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+10697%E2%80%9310707%2E+PMLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/)
36. Folgert Karsdorp and Lauren Fonteyn. 2019. Cultural entrenchment of folktales is encoded in language. Palgrave Communications, 5(1):25. [find](https://scholar.google.com/scholar?q=Folgert+Karsdorp+and+Lauren+Fonteyn%2E+2019%2E+Cultural+entrenchment+of+folktales+is+encoded+in+language%2E+Palgrave+Communications%2C+5%281%29%3A25%2E)
37. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2020%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+In+International+Conference+on+Learning+Repr)
38. Abhimanu Kumar, Matthew Lease, and Jason Baldridge. 2011. Supervised language modeling for temporal resolution of texts. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 2069–2072. [find](https://scholar.google.com/scholar?q=Abhimanu+Kumar%2C+Matthew+Lease%2C+and+Jason+Baldridge%2E+2011%2E+Supervised+language+modeling+for+temporal+resolution+of+texts%2E+In+Proceedings+of+the+20th+ACM+international+conference+on+Information+and+know)
39. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2022%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+o)
40. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Associ… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
41. Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Li+Lucy+and+David+Bamman%2E+2021%2E+Gender+and+representation+bias+in+GPT%2D3+generated+stories%2E+In+Proceedings+of+the+Third+Workshop+on+Narrative+Understanding%2C+pages+48%E2%80%9355%2C+Virtual%2E+Association+for+Comput) [our card](https://md.research.elamur.ai/free/en/papers/lucy-2021-gender-representation-bias-gpt-3/)
42. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+2022%2E+Data+contamination%3A+From+memorization+to+exploitation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+P)
43. Marc Marone and Benjamin Van Durme. 2023. Data portraits: Recording foundation model training data. [find](https://scholar.google.com/scholar?q=Marc+Marone+and+Benjamin+Van+Durme%2E+2023%2E+Data+portraits%3A+Recording+foundation+model+training+data%2E)
44. Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826, Abu… [find](https://scholar.google.com/scholar?q=Fatemehsadat+Mireshghallah%2C+Archit+Uniyal%2C+Tianhao+Wang%2C+David+Evans%2C+and+Taylor+Berg%2DKirkpatrick%2E+2022%2E+An+empirical+analysis+of+memorization+in+fine%2Dtuned+autoregressive+language+models%2E+In+Proceedi)
45. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Austin, Texas. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Takeshi+Onishi%2C+Hai+Wang%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+and+David+McAllester%2E+2016%2E+Who+did+what%3A+A+large%2Dscale+person%2Dcentered+cloze+dataset%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+)
46. Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 298–311, Online and Punta Cana, Dominican Republic. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Andrew+Piper%2C+Richard+Jean+So%2C+and+David+Bamman%2E+2021%2E+Narrative+theory+for+computational+narrative+understanding%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
48. Stephen Ramsay and Geoffrey Rockwell. 2012. Developing things: Notes toward an epistemology of building in the digital humanities. Debates in the digital humanities, pages 75–84. [find](https://scholar.google.com/scholar?q=Stephen+Ramsay+and+Geoffrey+Rockwell%2E+2012%2E+Developing+things%3A+Notes+toward+an+epistemology+of+building+in+the+digital+humanities%2E+Debates+in+the+digital+humanities%2C+pages+75%E2%80%9384%2E)
49. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840–854, Abu Dhabi, United Arab Emirates. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Yasaman+Razeghi%2C+Robert+L+Logan+IV%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2022%2E+Impact+of+pretraining+term+frequencies+on+few%2Dshot+numerical+reasoning%2E+In+Findings+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
50. Nicolas Ruth, Andreas Niekler, and Manuel Burghardt. 2022. Peeking Inside the DH Toolbox – Detection and Classification of Software Tools in DH Publications. Proceedings of the Computational Humanities Research Conference 2022, 1613:0073. [find](https://scholar.google.com/scholar?q=Nicolas+Ruth%2C+Andreas+Niekler%2C+and+Manuel+Burghardt%2E+2022%2E+Peeking+Inside+the+DH+Toolbox+%E2%80%93+Detection+and+Classification+of+Software+Tools+in+DH+Publications%2E+Proceedings+of+the+Computational+Humanitie)
51. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
52. Henning Schmidgen, Bernhard Dotzler, and Benno Stein. 2023. From the archive to the computer: Michel Foucault and the digital humanities. Journal of cultural analytics, 7(4). [find](https://scholar.google.com/scholar?q=Henning+Schmidgen%2C+Bernhard+Dotzler%2C+and+Benno+Stein%2E+2023%2E+From+the+archive+to+the+computer%3A+Michel+Foucault+and+the+digital+humanities%2E+Journal+of+cultural+analytics%2C+7%284%29%2E)
53. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE. [find](https://scholar.google.com/scholar?q=Reza+Shokri%2C+Marco+Stronati%2C+Congzheng+Song%2C+and+Vitaly+Shmatikov%2E+2017%2E+Membership+inference+attacks+against+machine+learning+models%2E+In+2017+IEEE+symposium+on+security+and+privacy+%28SP%29%2C+pages+3%E2%80%9318%2E+) [our card](https://md.research.elamur.ai/free/en/papers/shokri-2016-membership-inference-attacks-against-machine/)
54. Arthur Spirling. 2023. Why open-source generative AI models are an ethical way forward for science. Nature, 616(7957):413–413. [find](https://scholar.google.com/scholar?q=Arthur+Spirling%2E+2023%2E+Why+open%2Dsource+generative+AI+models+are+an+ethical+way+forward+for+science%2E+Nature%2C+616%287957%29%3A413%E2%80%93413%2E)
55. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56, Seattle, United States. Association for Computation… [find](https://scholar.google.com/scholar?q=Dominik+Stammbach%2C+Maria+Antoniak%2C+and+Elliott+Ash%2E+2022%2E+Heroes%2C+villains%2C+and+victims%2C+and+GPT%2D3%3A+Automated+extraction+of+character+roles+without+training+data%2E+In+Proceedings+of+the+4th+Workshop+of)
56. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p… [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Roy+Schwartz%2C+Nicholas+Lourie%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2020%2E+Dataset+cartography%3A+Mapping+and+diagnosing+datasets+with+training+dynamics%2E+)
57. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290. [find](https://scholar.google.com/scholar?q=Kushal+Tirumala%2C+Aram+Markosyan%2C+Luke+Zettlemoyer%2C+and+Armen+Aghajanyan%2E+2022%2E+Memorization+without+overfitting%3A+Analyzing+the+training+dynamics+of+large+language+models%2E+Advances+in+Neural+Informatio) [our card](https://md.research.elamur.ai/free/en/papers/tirumala-2022-memorization-overfitting-analyzing-training-dynamics/)
58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+2023%2E+Llama%3A+Open+and+efficient+f)
59. Ted Underwood. 2018. Why literary time is measured in minutes. ELH, 85(2). [find](https://scholar.google.com/scholar?q=Ted+Underwood%2E+2018%2E+Why+literary+time+is+measured+in+minutes%2E+ELH%2C+85%282%29%2E)
60. Ted Underwood. 2023. Using GPT-4 to measure the passage of time in fiction. [https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/](https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/). [↗](https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/)
61. Melanie Walsh and Maria Antoniak. 2021. The Goodreads “Classics”: A computational study of readers, Amazon, and crowdsourced amateur criticism. Journal of Cultural Analytics, 6(2). [find](https://scholar.google.com/scholar?q=Melanie+Walsh+and+Maria+Antoniak%2E+2021%2E+The+Goodreads+%E2%80%9CClassics%E2%80%9D%3A+A+computational+study+of+readers%2C+Amazon%2C+and+crowdsourced+amateur+criticism%2E+Journal+of+Cultural+Analytics%2C+6%282%29%2E)
62. Michael Yoder, Sopan Khosla, Qinlan Shen, Aakanksha Naik, Huiming Jin, Hariharan Muralidharan, and Carolyn Rosé. 2021. FanfictionNLP: A text processing pipeline for fanfiction. In Proceedings of the Third Workshop on Narrative Understanding, pages 13–23, Virtual. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Michael+Yoder%2C+Sopan+Khosla%2C+Qinlan+Shen%2C+Aakanksha+Naik%2C+Huiming+Jin%2C+Hariharan+Muralidharan%2C+and+Carolyn+Ros%C3%A9%2E+2021%2E+FanfictionNLP%3A+A+text+processing+pipeline+for+fanfiction%2E+In+Proceedings+of+the+T)
63. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938. — [arXiv:2112.12938](https://arxiv.org/abs/2112.12938) [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Matthew+Jagielski%2C+Florian+Tram%C3%A8r%2C+and+Nicholas+Carlini%2E+2021%2E+Counterfactual+memorization+in+neural+language+models%2E+arXiv+preprint+arXiv%3A2112%2E12938%2E+%E2%80%94+)
64. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+et+al%2E+2022%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models)
65. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
66. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? arXiv submission 4840038. [find](https://scholar.google.com/scholar?q=Caleb+Ziems%2C+William+Held%2C+Omar+Shaikh%2C+Jiaao+Chen%2C+Zhehao+Zhang%2C+and+Diyi+Yang%2E+2023%2E+Can+large+language+models+transform+computational+social+science%3F+arXiv+submission+4840038%2E) [our card](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/)