---
title: "Estimating Training Data Influence by Tracing Gradient Descent"
description: "The authors propose TracIn. 95.5% accuracy. 28 references, cited by 108 papers here. Among them: 115."
authors: ["Garima Pruthi", "Frederick Liu", "Mukund Sundararajan", "Satyen Kale"]
year: 2020
venue: "arXiv:2002.08484"
source: "https://arxiv.org/abs/2002.08484"
doi: "10.48550/arxiv.2002.08484"
arxiv: "2002.08484"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pruthi-2020-estimating-training-data-influence-tracing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pruthi-2020-estimating-training-data-influence-tracing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Estimating Training Data Influence by Tracing Gradient Descent

# Estimating Training Data Influence by Tracing Gradient Descent

**Authors:** Garima Pruthi, [Frederick Liu](https://md.research.elamur.ai/free/en/authors/frederick-liu/), [Mukund Sundararajan](https://md.research.elamur.ai/free/en/authors/mukund-sundararajan/) and [Satyen Kale](https://md.research.elamur.ai/free/en/authors/satyen-kale/) · **Year:** 2020 · **Venue:** arXiv:2002.08484 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2002.08484) · [PDF](https://arxiv.org/pdf/2002.08484) · [DOI 10.48550/arxiv.2002.08484](https://doi.org/10.48550/arxiv.2002.08484) · [arXiv 2002.08484](https://arxiv.org/abs/2002.08484) · [OpenAlex](https://openalex.org/W3101656801)

**Citations:** 115 citations (OpenAlex, 2026-10-14) · [108](https://md.research.elamur.ai/free/en/papers/pruthi-2020-estimating-training-data-influence-tracing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks which training examples are responsible for a model's prediction on a given input. The authors propose TracIn. It attributes to each training example the drop in the test example's loss that occurred during the optimization steps where that example was used. The exact version would require knowing the test point while training, so they derive a practical variant. A first-order expansion turns each step's contribution into a learning-rate-weighted dot product between the training-example gradient and the test-example gradient. Saved checkpoints stand in for the full training trajectory (TracInCP), and gradients can be restricted to selected layers to reduce cost. The method only needs gradients, checkpoints and a loss. It therefore applies to any model trained with SGD-like optimizers.

The authors compare it with influence functions and representer point selection, using self-influence to surface mislabelled training data. They also demonstrate it on regression, text and image tasks. The result is an easy-to-implement tool for auditing and cleaning training sets.

## Key points
- **Two variants:**
  - TracIn (first-order, per-step) sums η_t·∇ℓ(w_t, z')·∇ℓ(w_t, z) over the steps that used z, with averaging for minibatches.
  - TracInCP replaces per-step parameters with k stored checkpoints.
- **Evaluation:** training points are ranked by self-influence, and the metric is the share of mislabelled examples recovered in top prefixes. Baselines are influence functions and representer points.
- **DBPedia (text):** 560K training and 70K test examples across 14 classes, using a SWEM model with a sentencepiece tokenizer. The model reached about 95.5% accuracy. Influence was computed from 6 checkpoints on the last fully connected layer.
- **ImageNet:** ResNet-50 at 73% top-1, trained on 1.28M images in 1000 classes. TracInCP used checkpoints 30/60/90, with gradients projected to 1472 dimensions via a rank-1 trick for fully connected layers.
- **Housing-price regression:** a model with 168K parameters, 3 hidden layers and test explained variance of 0.70. High self-influence points came from densely populated areas, which the authors link to memorization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Garima Pruthi, Frederick Liu, Mukund Sundararajan and Satyen Kale. 2020. Estimating Training Data Influence by Tracing Gradient Descent. arXiv:2002.08484.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pruthi2020estimating,
  title = {{Estimating Training Data Influence by Tracing Gradient Descent}},
  author = {Garima Pruthi, Frederick Liu, Mukund Sundararajan and Satyen Kale},
  year = {2020},
  doi = {10.48550/arxiv.2002.08484},
  eprint = {2002.08484},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.08484}
}
```

- [← previous Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://md.research.elamur.ai/free/en/papers/mehri-2020-unsupervised-evaluation-interactive-dialog-dialogpt/)
- [next → SLURP: A Spoken Language Understanding Resource Package](https://md.research.elamur.ai/free/en/papers/bastianelli-2020-slurp-spoken-language-understanding-resource/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. TracIn Code. [https://github.com/frederick0329/TracIn](https://github.com/frederick0329/TracIn). [↗](https://github.com/frederick0329/TracIn)
2. Pang Wei Koh and Percy Liang. [Understanding black-box predictions via influence functions](https://md.research.elamur.ai/free/en/papers/koh-2017-understanding-black-box-predictions-influence/). In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1885–1894, 2017.
3. Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, pages 9291–9301, 2018. [find](https://scholar.google.com/scholar?q=Chih%2DKuan+Yeh%2C+Joon+Kim%2C+Ian+En%2DHsu+Yen%2C+and+Pradeep+K+Ravikumar%2E+Representer+point+selection+for+explaining+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+9291%E2%80%939301)
4. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328, 2017. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+Axiomatic+attribution+for+deep+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2DVolume+70%2C+pages+3319%E2%80%933328%2C+2017%2E)
5. Scott M Lundberg and Su-In Lee. [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In Advances in neural information processing systems, pages 4765–4774, 2017.
6. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135[–1144, 2016](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/).
7. Art B Owen and Clémentine Prieur. On shapley value for measuring importance of dependent inputs. SIAM/ASA Journal on Uncertainty Quantification, 5(1):986–1002, 2017. [find](https://scholar.google.com/scholar?q=Art+B+Owen+and+Cl%C3%A9mentine+Prieur%2E+On+shapley+value+for+measuring+importance+of+dependent+inputs%2E+SIAM%2FASA+Journal+on+Uncertainty+Quantification%2C+5%281%29%3A986%E2%80%931002%2C+2017%2E)
8. Art B Owen. Sobol’indices and shapley value. SIAM/ASA Journal on Uncertainty Quantification, 2(1):245–251, 2014. [find](https://scholar.google.com/scholar?q=Art+B+Owen%2E+Sobol%E2%80%99indices+and+shapley+value%2E+SIAM%2FASA+Journal+on+Uncertainty+Quantification%2C+2%281%29%3A245%E2%80%93251%2C+2014%2E)
9. Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167–1176, 2019. [find](https://scholar.google.com/scholar?q=Ruoxi+Jia%2C+David+Dao%2C+Boxin+Wang%2C+Frances+Ann+Hubis%2C+Nick+Hynes%2C+Nezihe+Merve+G%C3%BCrel%2C+Bo+Li%2C+Ce+Zhang%2C+Dawn+Song%2C+and+Costas+J+Spanos%2E+Towards+efficient+data+valuation+based+on+the+shapley+value%2E+In+Th)
10. Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242–2251, 2019. [find](https://scholar.google.com/scholar?q=Amirata+Ghorbani+and+James+Zou%2E+Data+shapley%3A+Equitable+valuation+of+data+for+machine+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages+2242%E2%80%932251%2C+2019%2E)
11. Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. In Advances in Neural Information Processing Systems, pages 4215–4224, 2019. [find](https://scholar.google.com/scholar?q=Satoshi+Hara%2C+Atsushi+Nitanda%2C+and+Takanori+Maehara%2E+Data+cleansing+for+models+trained+with+sgd%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+4215%E2%80%934224%2C+2019%2E)
12. Rico Sennrich, Barry Haddow, and Alexandra Birch. [Edinburgh neural machine translation systems for WMT 16](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/). In Proceedings of the First Conference on Machine Translation, pages 371–376, 2016.
13. Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, 2017. [find](https://scholar.google.com/scholar?q=Samuli+Laine+and+Timo+Aila%2E+Temporal+ensembling+for+semi%2Dsupervised+learning%2E+In+5th+International+Conference+on+Learning+Representations%2C+2017%2E)
14. Hao-Ran Wei, Shujian Huang, Ran Wang, Xin-Yu Dai, and Jiajun Chen. Online distilling from checkpoints for neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 17th Conference of the North American Chapter of the Association for Computational Li… [find](https://scholar.google.com/scholar?q=Hao%2DRan+Wei%2C+Shujian+Huang%2C+Ran+Wang%2C+Xin%2DYu+Dai%2C+and+Jiajun+Chen%2E+Online+distilling+from+checkpoints+for+neural+machine+translation%2E+In+Jill+Burstein%2C+Christy+Doran%2C+and+Thamar+Solorio%2C+editors%2C+Proc)
15. Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICML, 2019. [find](https://scholar.google.com/scholar?q=Yanyao+Shen+and+Sujay+Sanghavi%2E+Learning+with+bad+training+data+via+iterative+trimmed+loss+minimization%2E+In+ICML%2C+2019%2E)
16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
17. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Canada, 2009. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+University+of+Toronto%2C+Toronto%2C+Canada%2C+2009%2E)
18. R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291–297, 1997. [find](https://scholar.google.com/scholar?q=R+Kelley+Pace+and+Ronald+Barry%2E+Sparse+spatial+autoregressions%2E+Statistics+%26+Probability+Letters%2C+33%283%29%3A291%E2%80%93297%2C+1997%2E)
19. Xiang Zhang, Junbo Zhao, and Yann LeCun. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in neural information processing systems, pages 649–657, 2015.
20. Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. [Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms](https://md.research.elamur.ai/free/en/papers/shen-2018-baseline-needs-more-love-simple/). In Proceedings of the 56th Annual Meeting of the Association for C…
21. Taku Kudo and John Richardson. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, 2018.
22. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages+24)
23. Xiaochuang Han and Yulia Tsvetkov. Fortifying toxic speech detectors against veiled toxicity, 2020. [find](https://scholar.google.com/scholar?q=Xiaochuang+Han+and+Yulia+Tsvetkov%2E+Fortifying+toxic+speech+detectors+against+veiled+toxicity%2C+2020%2E)
24. R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982. [find](https://scholar.google.com/scholar?q=R+Dennis+Cook+and+Sanford+Weisberg%2E+Residuals+and+influence+in+regression%2E+New+York%3A+Chapman+and+Hall%2C+1982%2E)
25. Barak A. Pearlmutter. Fast exact multiplication by the hessian. Neural Computation, 6:147–160, 1994. [find](https://scholar.google.com/scholar?q=Barak+A%2E+Pearlmutter%2E+Fast+exact+multiplication+by+the+hessian%2E+Neural+Computation%2C+6%3A147%E2%80%93160%2C+1994%2E)
26. Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory, pages 416–426. Springer, 2001. [find](https://scholar.google.com/scholar?q=Bernhard+Sch%C3%B6lkopf%2C+Ralf+Herbrich%2C+and+Alex+J+Smola%2E+A+generalized+representer+theorem%2E+In+International+conference+on+computational+learning+theory%2C+pages+416%E2%80%93426%2E+Springer%2C+2001%2E)
27. Erik Bernhardsson. Annoy: Approximate Nearest Neighbors in C++/Python, 2018. Python package version 1.13.0. [find](https://scholar.google.com/scholar?q=Erik+Bernhardsson%2E+Annoy%3A+Approximate+Nearest+Neighbors+in+C%2B%2B%2FPython%2C+2018%2E+Python+package+version+1%2E13%2E0%2E)
28. David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1–2):1–157, 2014. [find](https://scholar.google.com/scholar?q=David+P+Woodruff+et+al%2E+Sketching+as+a+tool+for+numerical+linear+algebra%2E+Foundations+and+Trends%C2%AE+in+Theoretical+Computer+Science%2C+10%281%E2%80%932%29%3A1%E2%80%93157%2C+2014%2E)

*Updated 2 November 2026*