---
title: "Studying Large Language Model Generalization with Influence"
description: "The work offers a scalable way to study how LLMs generalize from their data. 111 references, cited by 79 papers here."
authors: ["Roger Grosse", "Juhan Bae", "Cem Anil", "Nelson Elhage", "Alex Tamkin", "Amirhossein Tajdini", "Benoit Steiner", "Dustin Li", "Esin Durmus", "Ethan Perez", "Evan Hubinger", "Kamile Luko\\vsiute", "Karina Nguyen", "Nicholas Joseph", "Sam McCandlish", "Jared Kaplan", "Samuel R. Bowman"]
year: 2023
venue: "arXiv:2308.03296"
source: "https://arxiv.org/abs/2308.03296"
doi: "10.48550/arxiv.2308.03296"
arxiv: "2308.03296"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/grosse-2023-studying-large-language-model-generalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/grosse-2023-studying-large-language-model-generalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Studying Large Language Model Generalization with Influence Functions

# Studying Large Language Model Generalization with Influence Functions

**Authors:** Roger Grosse, Juhan Bae, [Cem Anil](https://md.research.elamur.ai/free/en/authors/cem-anil/), [Nelson Elhage](https://md.research.elamur.ai/free/en/authors/nelson-elhage/), [Alex Tamkin](https://md.research.elamur.ai/free/en/authors/alex-tamkin/), Amirhossein Tajdini, [Benoit Steiner](https://md.research.elamur.ai/free/en/authors/benoit-steiner/), [Dustin Li](https://md.research.elamur.ai/free/en/authors/dustin-li/), [Esin Durmus](https://md.research.elamur.ai/free/en/authors/esin-durmus/), [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/), [Evan Hubinger](https://md.research.elamur.ai/free/en/authors/evan-hubinger/), [Kamile Luko\\vsiute](https://md.research.elamur.ai/free/en/authors/kamile-luko-vsiute/), [Karina Nguyen](https://md.research.elamur.ai/free/en/authors/karina-nguyen/), [Nicholas Joseph](https://md.research.elamur.ai/free/en/authors/nicholas-joseph/), [Sam McCandlish](https://md.research.elamur.ai/free/en/authors/sam-mccandlish/), [Jared Kaplan](https://md.research.elamur.ai/free/en/authors/jared-kaplan/) and [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2023 · **Venue:** arXiv:2308.03296 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.03296) · [PDF](https://arxiv.org/pdf/2308.03296) · [DOI 10.48550/arxiv.2308.03296](https://doi.org/10.48550/arxiv.2308.03296) · [arXiv 2308.03296](https://arxiv.org/abs/2308.03296) · [OpenAlex](https://openalex.org/W4385681731)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/grosse-2023-studying-large-language-model-generalization/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Knowing which training examples drive a given behavior in a large language model could help researchers anticipate and reduce its risks. Influence functions estimate how a model's parameters and outputs would shift if one sequence were added to its training data. However, they require an inverse-Hessian-vector product (IHVP), which has kept them impractical beyond small networks. Grosse and colleagues at Anthropic adapt the EK-FAC (Eigenvalue-corrected Kronecker-Factored Approximate Curvature) approximation to transformer language models with up to 52 billion parameters. When checked against the proximal Bregman response function (PBRF), EK-FAC is about as accurate as conventional estimators while computing IHVPs dramatically faster. To lower the cost of gradients over candidate training sequences, the authors add TF-IDF filtering and query batching. They also split influence across individual layers and tokens. The resulting tool shows sparse influence patterns and more abstract generalization in bigger models, along with cross-lingual, math, coding and role-playing effects. Yet influence nearly disappears when key phrases appear in reversed order. The work offers a scalable way to study how LLMs generalize from their data.

## Key points
- **Scale:** EK-FAC-based influence estimation is applied to LLMs of up to 52 billion parameters. An 810 million parameter model is also analyzed in the appendices (arXiv:2308.03296).
- **Validation:** EK-FAC is compared with PBRF and with traditional estimators. It reaches similar accuracy, and its IHVP step runs orders of magnitude faster.
- **Efficiency tricks:** TF-IDF filtering narrows the pool of candidate training sequences, and query batching shares gradient computation across many queries. Influence can also be attributed to specific layers and tokens.
- **Findings:** Influence is sparse, and generalization becomes more abstract with model scale. The paper also studies math and programming abilities, cross-lingual transfer, memorization and role-playing, and crowdworkers interpreted the top influential sequences.
- **Limitation found:** Reversing the sequence in which the key phrases appear causes influence to fall to almost zero. This points to a notable gap in how these models generalize.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Luko\\vsiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan and Samuel R. Bowman. 2023. Studying Large Language Model Generalization with Influence Functions. arXiv:2308.03296.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{grosse2023studying,
  title = {{Studying Large Language Model Generalization with Influence Functions}},
  author = {Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukovsiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan and Samuel R. Bowman},
  year = {2023},
  doi = {10.48550/arxiv.2308.03296},
  eprint = {2308.03296},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.03296}
}
```

- [← previous Statistical Rejection Sampling Improves Preference Optimization](https://md.research.elamur.ai/free/en/papers/liu-2023-statistical-rejection-sampling-improves-preference/)
- [next → Zero-Shot Listwise Document Reranking with a Large Language Model](https://md.research.elamur.ai/free/en/papers/ma-2023-zero-shot-listwise-document-reranking/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

111 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abubakar Abid, Maheen Farooqi, and James Zou. Large language models associate Muslims with violence. Nature Machine Intelligence, 3(6):461–463, 2021. [find](https://scholar.google.com/scholar?q=Abubakar+Abid%2C+Maheen+Farooqi%2C+and+James+Zou%2E+Large+language+models+associate+Muslims+with+violence%2E+Nature+Machine+Intelligence%2C+3%286%29%3A461%E2%80%93463%2C+2021%2E)
2. Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. The Journal of Machine Learning Research, 18(1): 4148–4187, 2017. [find](https://scholar.google.com/scholar?q=Naman+Agarwal%2C+Brian+Bullins%2C+and+Elad+Hazan%2E+Second%2Dorder+stochastic+optimization+for+machine+learning+in+linear+time%2E+The+Journal+of+Machine+Learning+Research%2C+18%281%29%3A+4148%E2%80%934187%2C+2017%2E)
3. Shun-Ichi Amari. Neural learning in structured parameter spaces - Natural Riemannian gradient. In Advances in Neural Information Processing Systems, 1996. [find](https://scholar.google.com/scholar?q=Shun%2DIchi+Amari%2E+Neural+learning+in+structured+parameter+spaces+%2D+Natural+Riemannian+gradient%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+1996%2E)
4. Jacob Andreas. [Language models as agent models](https://md.research.elamur.ai/free/en/papers/andreas-2022-language-models-as-agent-models/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5769–5779, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.findings-emnlp.423. [↗](https://aclanthology.org/2022.findings-emnlp.423)
5. Walter Edwin Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics, 9(1):17–29, 1951. [find](https://scholar.google.com/scholar?q=Walter+Edwin+Arnoldi%2E+The+principle+of+minimized+iterations+in+the+solution+of+the+matrix+eigenvalue+problem%2E+Quarterly+of+applied+mathematics%2C+9%281%29%3A17%E2%80%9329%2C+1951%2E)
6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+Nelson+Elhage%2C+Zac+Hatfield%2DDodds%2C+Danny+Hernandez%2C+Jackson+Kernion%2C)
7. Jimmy Ba, Roger Grosse, and James Martens. Distributed second-order optimization using Kronecker-factored approximations. In International Conference on Learning Representations, 2017. [find](https://scholar.google.com/scholar?q=Jimmy+Ba%2C+Roger+Grosse%2C+and+James+Martens%2E+Distributed+second%2Dorder+optimization+using+Kronecker%2Dfactored+approximations%2E+In+International+Conference+on+Learning+Representations%2C+2017%2E)
8. Juhan Bae, Guodong Zhang, and Roger Grosse. Eigenvalue corrected noisy natural gradient, 2018. [find](https://scholar.google.com/scholar?q=Juhan+Bae%2C+Guodong+Zhang%2C+and+Roger+Grosse%2E+Eigenvalue+corrected+noisy+natural+gradient%2C+2018%2E)
9. Juhan Bae, Nathan Hoyen Ng, Alston Lo, Marzyeh Ghassemi, and Roger Baker Grosse. If influence functions are the answer, then what is the question? In Advances in Neural Information Processing Systems, 2022a. [find](https://scholar.google.com/scholar?q=Juhan+Bae%2C+Nathan+Hoyen+Ng%2C+Alston+Lo%2C+Marzyeh+Ghassemi%2C+and+Roger+Baker+Grosse%2E+If+influence+functions+are+the+answer%2C+then+what+is+the+question%3F+In+Advances+in+Neural+Information+Processing+Systems%2C)
10. Juhan Bae, Paul Vicol, Jeff Z. HaoChen, and Roger Baker Grosse. Amortized proximal optimization. In Advances in Neural Information Processing Systems, 2022b. [find](https://scholar.google.com/scholar?q=Juhan+Bae%2C+Paul+Vicol%2C+Jeff+Z%2E+HaoChen%2C+and+Roger+Baker+Grosse%2E+Amortized+proximal+optimization%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022b%2E)
11. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
12. Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. RelatIF: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pages 1899–1909. [find](https://scholar.google.com/scholar?q=Elnaz+Barshan%2C+Marc%2DEtienne+Brunet%2C+and+Gintare+Karolina+Dziugaite%2E+RelatIF%3A+Identifying+explanatory+training+samples+via+relative+influence%2E+In+International+Conference+on+Artificial+Intelligence+and)
13. Samyadeep Basu, Xuchen You, and Soheil Feizi. On second-order group influence functions for black-box predictions. In International Conference on Machine Learning, pages 715–724. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Samyadeep+Basu%2C+Xuchen+You%2C+and+Soheil+Feizi%2E+On+second%2Dorder+group+influence+functions+for+black%2Dbox+predictions%2E+In+International+Conference+on+Machine+Learning%2C+pages+715%E2%80%93724%2E+PMLR%2C+2020%2E)
14. Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Samyadeep+Basu%2C+Phil+Pope%2C+and+Soheil+Feizi%2E+Influence+functions+in+deep+learning+are+fragile%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
15. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Conference on Fairness, Accountability, and Transparency, 2021. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Conference+on+Fairness%2C+Accountability%2C+and+)
16. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
17. Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014. [find](https://scholar.google.com/scholar?q=Nick+Bostrom%2E+Superintelligence%3A+Paths%2C+Dangers%2C+Strategies%2E+Oxford+University+Press%2C+2014%2E)
18. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
19. Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. Understanding the origins of bias in word embeddings. In International Conference on Machine Learning, pages 803–811. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Marc%2DEtienne+Brunet%2C+Colleen+Alkalay%2DHoulihan%2C+Ashton+Anderson%2C+and+Richard+Zemel%2E+Understanding+the+origins+of+bias+in+word+embeddings%2E+In+International+Conference+on+Machine+Learning%2C+pages+803%E2%80%93811%2E)
20. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In USENIX Security Symposium, 2021.
21. Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Charpiat%2C+Nicolas+Girard%2C+Loris+Felardos%2C+and+Yuliya+Tarabalka%2E+Input+similarity+from+the+neural+network+perspective%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
22. Bo Chen, Xingyi Cheng, Yangli-ao Geng, Shen Li, Xin Zeng, Boyan Wang, Jing Gong, Chiming Liu, Aohan Zeng, Yuxiao Dong, et al. xTrimoPGLM: Unified 100b-scale pre-trained transformer for deciphering the language of protein. bioRxiv, pages 2023–07, 2023. [find](https://scholar.google.com/scholar?q=Bo+Chen%2C+Xingyi+Cheng%2C+Yangli%2Dao+Geng%2C+Shen+Li%2C+Xin+Zeng%2C+Boyan+Wang%2C+Jing+Gong%2C+Chiming+Liu%2C+Aohan+Zeng%2C+Yuxiao+Dong%2C+et+al%2E+xTrimoPGLM%3A+Unified+100b%2Dscale+pre%2Dtrained+transformer+for+deciphering+the)
23. Brian Christian. The Alignment Problem. W. W. Norton & Company, 2020. [find](https://scholar.google.com/scholar?q=Brian+Christian%2E+The+Alignment+Problem%2E+W%2E+W%2E+Norton+%26+Company%2C+2020%2E)
24. Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661–703, 2009. [find](https://scholar.google.com/scholar?q=Aaron+Clauset%2C+Cosma+Rohilla+Shalizi%2C+and+Mark+EJ+Newman%2E+Power%2Dlaw+distributions+in+empirical+data%2E+SIAM+review%2C+51%284%29%3A661%E2%80%93703%2C+2009%2E)
25. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
26. R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982. [find](https://scholar.google.com/scholar?q=R+Dennis+Cook+and+Sanford+Weisberg%2E+Residuals+and+influence+in+regression%2E+New+York%3A+Chapman+and+Hall%2C+1982%2E)
27. Nikita Dhawan, Sicong Huang, Juhan Bae, and Roger Baker Grosse. Efficient parametric approximations of neural network function space distance. In International Conference on Machine Learning, pages 7795–7812. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Nikita+Dhawan%2C+Sicong+Huang%2C+Juhan+Bae%2C+and+Roger+Baker+Grosse%2E+Efficient+parametric+approximations+of+neural+network+function+space+distance%2E+In+International+Conference+on+Machine+Learning%2C+pages+77)
28. Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL [http://archive.ics.uci.edu/ml](http://archive.ics.uci.edu/ml). [↗](http://archive.ics.uci.edu/ml)
29. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
30. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circu… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+Roger+Grosse%2C+Sam+McCandlish%2C+Jared+Kaplan%2C+D)
31. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful AI: Developing and governing AI that does not lie, 2021. [find](https://scholar.google.com/scholar?q=Owain+Evans%2C+Owen+Cotton%2DBarratt%2C+Lukas+Finnveden%2C+Adam+Bales%2C+Avital+Balwit%2C+Peter+Wills%2C+Luca+Righetti%2C+and+William+Saunders%2E+Truthful+AI%3A+Developing+and+governing+AI+that+does+not+lie%2C+2021%2E)
32. Minghong Fang, Neil Zhenqiang Gong, and Jia Liu. Influence function based data poisoning attacks to top-n recommender systems. In Proceedings of The Web Conference, 2020. [find](https://scholar.google.com/scholar?q=Minghong+Fang%2C+Neil+Zhenqiang+Gong%2C+and+Jia+Liu%2E+Influence+function+based+data+poisoning+attacks+to+top%2Dn+recommender+systems%2E+In+Proceedings+of+The+Web+Conference%2C+2020%2E)
33. Vitaly Feldman and Chiyuan Zhang. [What neural networks memorize and why: Discovering the long tail via influence estimation](https://md.research.elamur.ai/free/en/papers/feldman-2020-neural-networks-memorize-why-discovering/). In Advances in Neural Information Processing Systems, 2020.
34. Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. [What can transformers learn in-context? A case study of simple function classes](https://md.research.elamur.ai/free/en/papers/garg-2022-transformers-learn-context-case-study/). In Advances in Neural Information Processing Systems, 2022.
35. Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a Kronecker-factored eigenbasis. In Advances in Neural Information Processing Systems, 2018. [find](https://scholar.google.com/scholar?q=Thomas+George%2C+C%C3%A9sar+Laurent%2C+Xavier+Bouthillier%2C+Nicolas+Ballas%2C+and+Pascal+Vincent%2E+Fast+approximate+natural+gradient+descent+in+a+Kronecker%2Dfactored+eigenbasis%2E+In+Advances+in+Neural+Information+Pr)
36. Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242–2251. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Amirata+Ghorbani+and+James+Zou%2E+Data+Shapley%3A+Equitable+valuation+of+data+for+machine+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages+2242%E2%80%932251%2E+PMLR%2C+2019%2E)
37. Roger Grosse and James Martens. A Kronecker-factored approximate Fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573–582. PMLR, 2016. [find](https://scholar.google.com/scholar?q=Roger+Grosse+and+James+Martens%2E+A+Kronecker%2Dfactored+approximate+Fisher+matrix+for+convolution+layers%2E+In+International+Conference+on+Machine+Learning%2C+pages+573%E2%80%93582%2E+PMLR%2C+2016%2E)
38. Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10333– 10350, Online and Punta Cana, Dominican… [find](https://scholar.google.com/scholar?q=Han+Guo%2C+Nazneen+Rajani%2C+Peter+Hase%2C+Mohit+Bansal%2C+and+Caiming+Xiong%2E+FastIF%3A+Scalable+influence+functions+for+efficient+model+interpretation+and+debugging%2E+In+Proceedings+of+the+2021+Conference+on+Em)
39. Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs, 2023. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Albert+Webson%2C+Ellie+Pavlick%2C+Lucas+Dixon%2C+Ian+Tenney%2C+and+Tolga+Bolukbasi%2E+Simfluence%3A+Modeling+the+influence+of+individual+training+examples+by+simulating+training+runs%2C+2023%2E)
40. Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey, 2023. [find](https://scholar.google.com/scholar?q=Zayd+Hammoudeh+and+Daniel+Lowd%2E+Training+data+influence+analysis+and+estimation%3A+A+survey%2C+2023%2E)
41. Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383–393, 1974. [find](https://scholar.google.com/scholar?q=Frank+R+Hampel%2E+The+influence+curve+and+its+role+in+robust+estimation%2E+Journal+of+the+american+statistical+association%2C+69%28346%29%3A383%E2%80%93393%2C+1974%2E)
42. Xiaochuang Han and Yulia Tsvetkov. Influence tuning: Demoting spurious correlations via instance attribution and instance-driven updates. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4398–4409, Punta Cana, Dominican Republic, November 2021. Association for Computat… [find](https://scholar.google.com/scholar?q=Xiaochuang+Han+and+Yulia+Tsvetkov%2E+Influence+tuning%3A+Demoting+spurious+correlations+via+instance+attribution+and+instance%2Ddriven+updates%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+)
43. Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. [Explaining black box predictions and unveiling data artifacts through influence functions](https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5553–5563, Online, July 2020. Association for Computation…
44. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
45. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2021. [find](https://scholar.google.com/scholar?q=Evan+Hubinger%2C+Chris+van+Merwijk%2C+Vladimir+Mikulik%2C+Joar+Skalse%2C+and+Scott+Garrabrant%2E+Risks+from+learned+optimization+in+advanced+machine+learning+systems%2C+2021%2E)
46. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. [Social biases in NLP models as barriers for persons with disabilities](https://md.research.elamur.ai/free/en/papers/hutchinson-2020-social-biases-nlp-models-as/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online, jul 2020. As…
47. Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Understanding predictions with data and data with predictions. In International Conference on Machine Learning, pages 9525–9587. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Andrew+Ilyas%2C+Sung+Min+Park%2C+Logan+Engstrom%2C+Guillaume+Leclerc%2C+and+Aleksander+Madry%2E+Datamodels%3A+Understanding+predictions+with+data+and+data+with+predictions%2E+In+International+Conference+on+Machine+)
48. Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3104–3122, 2021. [find](https://scholar.google.com/scholar?q=Matthew+Jagielski%2C+Giorgio+Severi%2C+Niklas+Pousette+Harger%2C+and+Alina+Oprea%2E+Subpopulation+data+poisoning+attacks%2E+In+Proceedings+of+the+2021+ACM+SIGSAC+Conference+on+Computer+and+Communications+Securi)
49. Janus. Simulators. LessWrong online forum, 2022. URL [https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators). [↗](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators)
50. Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the Shapley value. In International Conference on Artificial Intelligence and Statistics, pages 1167–1176. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Ruoxi+Jia%2C+David+Dao%2C+Boxin+Wang%2C+Frances+Ann+Hubis%2C+Nick+Hynes%2C+Nezihe+Merve+G%C3%BCrel%2C+Bo+Li%2C+Ce+Zhang%2C+Dawn+Song%2C+and+Costas+J+Spanos%2E+Towards+efficient+data+valuation+based+on+the+Shapley+value%2E+In+In)
51. Jeff Johnson, Matthijs Douze, and Hervé Jégou. [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). IEEE Transactions on Big Data, 7(3):535–547, 2019.
52. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+)
53. Diederick P Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In International Conference on Learning Representations, 2015.
54. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885–1894. [find](https://scholar.google.com/scholar?q=Pang+Wei+Koh+and+Percy+Liang%2E+Understanding+black%2Dbox+predictions+via+influence+functions%2E+In+International+Conference+on+Machine+Learning%2C+pages+1885%E2%80%931894%2E)
55. Pang Wei Koh, Kai-Siang Ang, Hubert Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Pang+Wei+Koh%2C+Kai%2DSiang+Ang%2C+Hubert+Teo%2C+and+Percy+Liang%2E+On+the+accuracy+of+influence+functions+for+measuring+group+effects%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
56. Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence-based data relabeling. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Shuming+Kong%2C+Yanyan+Shen%2C+and+Linpeng+Huang%2E+Resolving+training+biases+via+influence%2Dbased+data+relabeling%2E+In+International+Conference+on+Learning+Representations%2C+2021%2E)
57. Steven George Krantz and Harold R Parks. The Implicit Function Theorem: History, theory, and applications. Springer Science & Business Media, 2002. [find](https://scholar.google.com/scholar?q=Steven+George+Krantz+and+Harold+R+Parks%2E+The+Implicit+Function+Theorem%3A+History%2C+theory%2C+and+applications%2E+Springer+Science+%26+Business+Media%2C+2002%2E)
58. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+University+of+Toronto%2C+2009%2E)
59. Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Frederik+Kunstner%2C+Philipp+Hennig%2C+and+Lukas+Balles%2E+Limitations+of+the+empirical+Fisher+approximation+for+natural+gradient+descent%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2019%2E)
60. Faisal Ladhak, Esin Durmus, and Tatsunori Hashimoto. Contrastive error attribution for finetuned language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11482–11498, Toronto, Canada, July 2023. Association for Computa… [find](https://scholar.google.com/scholar?q=Faisal+Ladhak%2C+Esin+Durmus%2C+and+Tatsunori+Hashimoto%2E+Contrastive+error+attribution+for+finetuned+language+models%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguis)
61. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+L%C3%A9on+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
62. Donghoon Lee, Hyunsin Park, Trung Pham, and Chang D Yoo. Learning augmentation network via influence functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10961–10970, 2020. [find](https://scholar.google.com/scholar?q=Donghoon+Lee%2C+Hyunsin+Park%2C+Trung+Pham%2C+and+Chang+D+Yoo%2E+Learning+augmentation+network+via+influence+functions%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pa)
63. Stephanie Lin, Jacob Hilton, and Owain Evans. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, may 2022. Association for Computational Linguis…
64. Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Bingbin+Liu%2C+Jordan+T%2E+Ash%2C+Surbhi+Goel%2C+Akshay+Krishnamurthy%2C+and+Cyril+Zhang%2E+Transformers+learn+shortcuts+to+automata%2E+In+International+Conference+on+Learning+Representations%2C+2023%2E)
65. James Martens. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research, 21(1):5776–5851, 2020. [find](https://scholar.google.com/scholar?q=James+Martens%2E+New+insights+and+perspectives+on+the+natural+gradient+method%2E+The+Journal+of+Machine+Learning+Research%2C+21%281%29%3A5776%E2%80%935851%2C+2020%2E)
66. James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning, pages 2408–2417. PMLR, 2015. [find](https://scholar.google.com/scholar?q=James+Martens+and+Roger+Grosse%2E+Optimizing+neural+networks+with+Kronecker%2Dfactored+approximate+curvature%2E+In+International+Conference+on+Machine+Learning%2C+pages+2408%E2%80%932417%2E+PMLR%2C+2015%2E)
67. James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=James+Martens%2C+Jimmy+Ba%2C+and+Matt+Johnson%2E+Kronecker%2Dfactored+curvature+approximations+for+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E)
68. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
69. Richard Ngo, Lawrence Chan, and Sören Mindermann. [The alignment problem from a deep learning perspective](https://md.research.elamur.ai/free/en/papers/ngo-2022-alignment-problem-deep-learning-perspective/), 2022.
70. Elisa Nguyen, Minjoon Seo, and Seong Joon Oh. A Bayesian perspective on training data attribution, 2023. [find](https://scholar.google.com/scholar?q=Elisa+Nguyen%2C+Minjoon+Seo%2C+and+Seong+Joon+Oh%2E+A+Bayesian+perspective+on+training+data+attribution%2C+2023%2E)
71. Sejoon Oh, Sungchul Kim, Ryan A Rossi, and Srijan Kumar. Influence-guided data augmentation for neural tensor completion. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1386–1395, 2021. [find](https://scholar.google.com/scholar?q=Sejoon+Oh%2C+Sungchul+Kim%2C+Ryan+A+Rossi%2C+and+Srijan+Kumar%2E+Influence%2Dguided+data+augmentation+for+neural+tensor+completion%2E+In+Proceedings+of+the+30th+ACM+International+Conference+on+Information+%26+Knowl)
72. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… — [In-context Learning and Induction Heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/)
73. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
74. Kazuki Osawa, Shigang Li, and Torsten Hoefler. PipeFisher: Efficient training of large language models using pipelining and Fisher information matrices. Proceedings of Machine Learning and Systems, 5, 2023. [find](https://scholar.google.com/scholar?q=Kazuki+Osawa%2C+Shigang+Li%2C+and+Torsten+Hoefler%2E+PipeFisher%3A+Efficient+training+of+large+language+models+using+pipelining+and+Fisher+information+matrices%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%2C)
75. J Gregory Pauloski, Qi Huang, Lei Huang, Shivaram Venkataraman, Kyle Chard, Ian Foster, and Zhao Zhang. Kaisa: An adaptive second-order optimizer framework for deep neural networks. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages… [find](https://scholar.google.com/scholar?q=J+Gregory+Pauloski%2C+Qi+Huang%2C+Lei+Huang%2C+Shivaram+Venkataraman%2C+Kyle+Chard%2C+Ian+Foster%2C+and+Zhao+Zhang%2E+Kaisa%3A+An+adaptive+second%2Dorder+optimizer+framework+for+deep+neural+networks%2E+In+Proceedings+of+)
76. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. [Red teaming language models with language models](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448, Abu Dhabi,…
77. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain,… — [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
78. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. [Grokking: Generalization beyond overfitting on small algorithmic datasets](https://md.research.elamur.ai/free/en/papers/power-2022-grokking-generalization-overfitting-small-algorithmic/), 2022.
79. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. [Estimating training data influence by tracing gradient descent](https://md.research.elamur.ai/free/en/papers/pruthi-2020-estimating-training-data-influence-tracing/). In Advances in Neural Information Processing Systems, 2020.
80. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
81. Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. Explaining and improving model behavior with k nearest neighbor representations, 2020. [find](https://scholar.google.com/scholar?q=Nazneen+Fatema+Rajani%2C+Ben+Krause%2C+Wengpeng+Yin%2C+Tong+Niu%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+Explaining+and+improving+model+behavior+with+k+nearest+neighbor+representations%2C+2020%2E)
82. Juan Ramos. Using TF-IDF to determine word relevance in document queries. In Proceedings of the First Instructional Conference on Machine Learning, volume 242, pages 29–48. Citeseer, 2003. [find](https://scholar.google.com/scholar?q=Juan+Ramos%2E+Using+TF%2DIDF+to+determine+word+relevance+in+document+queries%2E+In+Proceedings+of+the+First+Instructional+Conference+on+Machine+Learning%2C+volume+242%2C+pages+29%E2%80%9348%2E+Citeseer%2C+2003%2E)
83. Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Rep… [find](https://scholar.google.com/scholar?q=Hubert+Ramsauer%2C+Bernhard+Sch%C3%A4fl%2C+Johannes+Lehner%2C+Philipp+Seidl%2C+Michael+Widrich%2C+Lukas+Gruber%2C+Markus+Holzleitner%2C+Thomas+Adler%2C+David+Kreil%2C+Michael+K+Kopp%2C+G%C3%BCnter+Klambauer%2C+Johannes+Brandstetter%2C)
84. Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In International Conference on Representation Learning, 2018. [find](https://scholar.google.com/scholar?q=Hippolyt+Ritter%2C+Aleksandar+Botev%2C+and+David+Barber%2E+A+scalable+Laplace+approximation+for+neural+networks%2E+In+International+Conference+on+Representation+Learning%2C+2018%2E)
85. Kevin Roose. A conversation with Bing’s chatbot left me deeply unsettled. The New York Times, 2023. [find](https://scholar.google.com/scholar?q=Kevin+Roose%2E+A+conversation+with+Bing%E2%80%99s+chatbot+left+me+deeply+unsettled%2E+The+New+York+Times%2C+2023%2E)
86. Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin Books, 2019. [find](https://scholar.google.com/scholar?q=Stuart+Russell%2E+Human+Compatible%3A+Artificial+Intelligence+and+the+Problem+of+Control%2E+Penguin+Books%2C+2019%2E)
87. Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8179–8186, 2022. [find](https://scholar.google.com/scholar?q=Andrea+Schioppa%2C+Polina+Zablotskaia%2C+David+Vilar%2C+and+Artem+Sokolov%2E+Scaling+up+influence+functions%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+36%2C+pages+8179%E2%80%938186%2C+2022%2E)
88. Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. [Linear transformers are secretly fast weight programmers](https://md.research.elamur.ai/free/en/papers/schlag-2021-linear-transformers-secretly-fast-weight/). In International Conference on Machine Learning, pages 9355–9366. PMLR, 2021.
89. Murray Shanahan, Kyle McDonell, and Laria Reynolds. [Role-play with large language models](https://md.research.elamur.ai/free/en/papers/shanahan-2023-role-play-large-language-models/), 2023.
90. Lloyd S Shapley. A value for n-person games. Classics in game theory, 69, 1997. [find](https://scholar.google.com/scholar?q=Lloyd+S+Shapley%2E+A+value+for+n%2Dperson+games%2E+Classics+in+game+theory%2C+69%2C+1997%2E)
91. Jonathan Richard Shewchuk. An introduction to the conjugate gradient method without the agonizing pain, 1994. [find](https://scholar.google.com/scholar?q=Jonathan+Richard+Shewchuk%2E+An+introduction+to+the+conjugate+gradient+method+without+the+agonizing+pain%2C+1994%2E)
92. Emily H. Soice, Rafael Rocha, Kimberlee Cordova, Michael Specter, and Kevin M. Esvelt. Can large language models democratize access to dual-use biotechnology?, 2023. [find](https://scholar.google.com/scholar?q=Emily+H%2E+Soice%2C+Rafael+Rocha%2C+Kimberlee+Cordova%2C+Michael+Specter%2C+and+Kevin+M%2E+Esvelt%2E+Can+large+language+models+democratize+access+to+dual%2Duse+biotechnology%3F%2C+2023%2E)
93. Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878, 2018. [find](https://scholar.google.com/scholar?q=Daniel+Soudry%2C+Elad+Hoffer%2C+Mor+Shpigel+Nacson%2C+Suriya+Gunasekar%2C+and+Nathan+Srebro%2E+The+implicit+bias+of+gradient+descent+on+separable+data%2E+The+Journal+of+Machine+Learning+Research%2C+19%281%29%3A2822%E2%80%932878%2C)
94. Zachary D Stephens, Skylar Y Lee, Faraz Faghri, Roy H Campbell, Chengxiang Zhai, Miles J Efron, Ravishankar Iyer, Michael C Schatz, Saurabh Sinha, and Gene E Robinson. Big data: Astronomical or genomical? PLoS biology, 13(7):e1002195, 2015. [find](https://scholar.google.com/scholar?q=Zachary+D+Stephens%2C+Skylar+Y+Lee%2C+Faraz+Faghri%2C+Roy+H+Campbell%2C+Chengxiang+Zhai%2C+Miles+J+Efron%2C+Ravishankar+Iyer%2C+Michael+C+Schatz%2C+Saurabh+Sinha%2C+and+Gene+E+Robinson%2E+Big+data%3A+Astronomical+or+genomi)
95. Zedong Tang, Fenlong Jiang, Maoguo Gong, Hao Li, Yue Wu, Fan Yu, Zidong Wang, and Min Wang. SKFAC: Training neural networks with faster Kronecker-factored approximate curvature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13479–13487, 2021. [find](https://scholar.google.com/scholar?q=Zedong+Tang%2C+Fenlong+Jiang%2C+Maoguo+Gong%2C+Hao+Li%2C+Yue+Wu%2C+Fan+Yu%2C+Zidong+Wang%2C+and+Min+Wang%2E+SKFAC%3A+Training+neural+networks+with+faster+Kronecker%2Dfactored+approximate+curvature%2E+In+Proceedings+of+the+)
96. Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive label cleaning with example-based explanations. In Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Stefano+Teso%2C+Andrea+Bontempelli%2C+Fausto+Giunchiglia%2C+and+Andrea+Passerini%2E+Interactive+label+cleaning+with+example%2Dbased+explanations%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2021%2E)
97. Athanasios Tsanas and Angeliki Xifara. Energy efficiency. UCI Machine Learning Repository, 2012. [find](https://scholar.google.com/scholar?q=Athanasios+Tsanas+and+Angeliki+Xifara%2E+Energy+efficiency%2E+UCI+Machine+Learning+Repository%2C+2012%2E)
98. Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins. Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence, 4(3):189–191, 2022. [find](https://scholar.google.com/scholar?q=Fabio+Urbina%2C+Filippa+Lentzos%2C+C%C3%A9dric+Invernizzi%2C+and+Sean+Ekins%2E+Dual+use+of+artificial%2Dintelligence%2Dpowered+drug+discovery%2E+Nature+Machine+Intelligence%2C+4%283%29%3A189%E2%80%93191%2C+2022%2E)
99. Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. [Transformers learn in-context by gradient descent](https://md.research.elamur.ai/free/en/papers/oswald-2022-transformers-learn-context-gradient-descent/). In International Conference on Machine Learning, pages 35151–35174. PMLR, 2023.
100. Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. EigenDamage: Structured pruning in the Kronecker-factored eigenbasis. In International Conference on Machine Learning, pages 6566–6575. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Chaoqi+Wang%2C+Roger+Grosse%2C+Sanja+Fidler%2C+and+Guodong+Zhang%2E+EigenDamage%3A+Structured+pruning+in+the+Kronecker%2Dfactored+eigenbasis%2E+In+International+Conference+on+Machine+Learning%2C+pages+6566%E2%80%936575%2E+PMLR)
101. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems, 2022.
102. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean L… — [Ethical and social risks of harm from Language Models](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
103. Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pages 11080–11090. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Gail+Weiss%2C+Yoav+Goldberg%2C+and+Eran+Yahav%2E+Thinking+like+transformers%2E+In+International+Conference+on+Machine+Learning%2C+pages+11080%E2%80%9311090%2E+PMLR%2C+2021%2E)
104. Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms, 2017. [find](https://scholar.google.com/scholar?q=Han+Xiao%2C+Kashif+Rasul%2C+and+Roland+Vollgraf%2E+Fashion%2DMNIST%3A+A+novel+image+dataset+for+benchmarking+machine+learning+algorithms%2C+2017%2E)
105. Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, 2018. [find](https://scholar.google.com/scholar?q=Chih%2DKuan+Yeh%2C+Joon+Kim%2C+Ian+En%2DHsu+Yen%2C+and+Pradeep+K+Ravikumar%2E+Representer+point+selection+for+explaining+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2018%2E)
106. Chih-Kuan Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. First is better than last for language data influence. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Chih%2DKuan+Yeh%2C+Ankur+Taly%2C+Mukund+Sundararajan%2C+Frederick+Liu%2C+and+Pradeep+Ravikumar%2E+First+is+better+than+last+for+language+data+influence%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
107. I-Cheng Yeh. Concrete compressive strength. UCI Machine Learning Repository, 2007. [find](https://scholar.google.com/scholar?q=I%2DCheng+Yeh%2E+Concrete+compressive+strength%2E+UCI+Machine+Learning+Repository%2C+2007%2E)
108. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models, 2021. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Daphne+Ippolito%2C+Katherine+Lee%2C+Matthew+Jagielski%2C+Florian+Tram%C3%A8r%2C+and+Nicholas+Carlini%2E+Counterfactual+memorization+in+neural+language+models%2C+2021%2E)
109. Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference on Machine Learning, pages 5852–5861. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Guodong+Zhang%2C+Shengyang+Sun%2C+David+Duvenaud%2C+and+Roger+Grosse%2E+Noisy+natural+gradient+as+variational+inference%2E+In+International+Conference+on+Machine+Learning%2C+pages+5852%E2%80%935861%2E+PMLR%2C+2018%2E)
110. Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems, 2019. [find](https://scholar.google.com/scholar?q=Guodong+Zhang%2C+Lala+Li%2C+Zachary+Nado%2C+James+Martens%2C+Sushant+Sachdeva%2C+George+Dahl%2C+Chris+Shallue%2C+and+Roger+B+Grosse%2E+Which+algorithmic+choices+matter+at+which+batch+sizes%3F+insights+from+a+noisy+quad)
111. Rui Zhang and Shihua Zhang. Rethinking influence functions of neural networks in the overparameterized regime. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9082–9090, 2022. [find](https://scholar.google.com/scholar?q=Rui+Zhang+and+Shihua+Zhang%2E+Rethinking+influence+functions+of+neural+networks+in+the+overparameterized+regime%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+36%2C+pages+9082%E2%80%939)

*Updated 2 November 2026*