---
title: "Scaling Laws and Interpretability of Learning from Repeated Data — Hernandez 2022"
description: "The authors from Anthropic investigate why even a small fraction of repeated data harms models so severely. 18 references, cited by 62 papers here."
authors: ["Danny Hernandez", "Tom Brown", "Tom Conerly", "Nova DasSarma", "Dawn Drain", "Sheer El-Showk", "Nelson Elhage", "Zac Hatfield-Dodds", "Tom Henighan", "Tristan Hume", "Scott Johnston", "Ben Mann", "Chris Olah", "Catherine Olsson", "Dario Amodei", "Nicholas Joseph", "Jared Kaplan", "Sam McCandlish"]
year: 2022
venue: "arXiv:2205.10487"
source: "https://arxiv.org/abs/2205.10487"
doi: "10.48550/arxiv.2205.10487"
arxiv: "2205.10487"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Scaling Laws and Interpretability of Learning from Repeated Data

# Scaling Laws and Interpretability of Learning from Repeated Data

**Authors:** [Danny Hernandez](https://md.research.elamur.ai/free/en/authors/danny-hernandez/), [Tom Brown](https://md.research.elamur.ai/free/en/authors/tom-brown/), Tom Conerly, Nova DasSarma, [Dawn Drain](https://md.research.elamur.ai/free/en/authors/dawn-drain/), Sheer El-Showk, Nelson Elhage, [Zac Hatfield-Dodds](https://md.research.elamur.ai/free/en/authors/zac-hatfield-dodds/), [Tom Henighan](https://md.research.elamur.ai/free/en/authors/tom-henighan/), Tristan Hume, [Scott Johnston](https://md.research.elamur.ai/free/en/authors/scott-johnston/), [Ben Mann](https://md.research.elamur.ai/free/en/authors/ben-mann/), Chris Olah, [Catherine Olsson](https://md.research.elamur.ai/free/en/authors/catherine-olsson/), [Dario Amodei](https://md.research.elamur.ai/free/en/authors/dario-amodei/), [Nicholas Joseph](https://md.research.elamur.ai/free/en/authors/nicholas-joseph/), [Jared Kaplan](https://md.research.elamur.ai/free/en/authors/jared-kaplan/) and [Sam McCandlish](https://md.research.elamur.ai/free/en/authors/sam-mccandlish/) · **Year:** 2022 · **Venue:** arXiv:2205.10487 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.10487) · [PDF](https://arxiv.org/pdf/2205.10487) · [DOI 10.48550/arxiv.2205.10487](https://doi.org/10.48550/arxiv.2205.10487) · [arXiv 2205.10487](https://arxiv.org/abs/2205.10487)

**Citations:** [62](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Large language models often see the same data several times. Sometimes this is done deliberately, to upweight high-quality sources, and sometimes it results from incomplete deduplication. The authors from Anthropic investigate why even a small fraction of repeated data harms models so severely. They train a family of transformer models on 100B tokens. Most of the data is unique, while a small subset is repeated many times. Across runs, they vary the model size, the fraction of repeated tokens, and the number of repetitions. The results are analyzed from two angles: through scaling laws and through mechanistic interpretability. The main finding is a pronounced double descent. The harm is greatest in a certain intermediate range of repetition frequency, roughly where the model begins to fully memorize the repeated subset. Copying and induction heads, that is, mechanisms associated with generalization, suffer the most. This leads to the hypothesis that memorization displaces generalization. In practice, this means that high-quality corpora such as Wikipedia or books should be duplicated with caution. The loss on the repeated subset can be monitored as a signal that repetition has already become harmful.

## Key points
- Data setup: 90% of tokens are unique, 10% are repetitions of a small subset. Model sizes range from roughly 1.5M to 805M parameters. Variants with 50% and 90% repeated data are also considered.
- An 800M-parameter model trained on 0.1% of the data repeated 100 times degrades to the level of a model half its size (roughly 340–400M).
- Extrapolation suggests that for GPT-3-scale models, noticeable harm may arise from repeating data as few as two times.
- To test copying, the authors used the opening paragraph of Harry Potter, repeated 11 times. With 3% repeated data, the effective model size on this test dropped by up to a factor of 3. On the prefix matching score, the drop averaged 32%, while on ordinary test loss it was no more than 15%.
- The effect replicates on Python code and on attention-only models with 1–2 layers. Pretraining on repeated data yields worse results than training from scratch.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/)

## Cite

Cite as: Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan and Sam McCandlish. 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv:2205.10487.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hernandez2022scaling,
  title = {{Scaling Laws and Interpretability of Learning from Repeated Data}},
  author = {Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan and Sam McCandlish},
  year = {2022},
  doi = {10.48550/arxiv.2205.10487},
  eprint = {2205.10487},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.10487}
}
```

- [← previous It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books](https://md.research.elamur.ai/free/en/papers/yao-2022-it-is-ai-s-turn/)
- [next → Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Advani, M. S. and Saxe, A. M. (2017). High-dimensional dynamics of generalization error in neural networks. [find](https://scholar.google.com/scholar?q=Advani%2C+M%2E+S%2E+and+Saxe%2C+A%2E+M%2E+%282017%29%2E+High%2Ddimensional+dynamics+of+generalization+error+in+neural+networks%2E)
2. Amodei, D., Hernandez, D., Sastry, G., Clark, J., Brockman, G., and Sutskever, I. (2018). Ai and compute. Heruntergeladen von [https://blog.openai](https://blog.openai). com/aiand-compute. [↗](https://blog.openai)
3. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general langua… [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+DasSarma%2C+N%2E%2C+Elhage%2C+N%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Hernandez%2C+D%2E%2C+Kernion%2C+J%2E%2C+Ndousse%2C+K%2E%2C+Olsson%2C+C%2E%2C)
4. Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2018). Reconciling modern machine learning practice and the bias-variance trade-off. [find](https://scholar.google.com/scholar?q=Belkin%2C+M%2E%2C+Hsu%2C+D%2E%2C+Ma%2C+S%2E%2C+and+Mandal%2C+S%2E+%282018%29%2E+Reconciling+modern+machine+learning+practice+and+the+bias%2Dvariance+trade%2Doff%2E)
5. Bi, B., Li, C., Wu, C., Yan, M., Wang, W., Huang, S., Huang, F., and Si, L. (2020). Palm: Pre-training an autoencoding and autoregressive language model for context-conditioned generation. [find](https://scholar.google.com/scholar?q=Bi%2C+B%2E%2C+Li%2C+C%2E%2C+Wu%2C+C%2E%2C+Yan%2C+M%2E%2C+Wang%2C+W%2E%2C+Huang%2C+S%2E%2C+Huang%2C+F%2E%2C+and+Si%2C+L%2E+%282020%29%2E+Palm%3A+Pre%2Dtraining+an+autoencoding+and+autoregressive+language+model+for+context%2Dconditioned+generation%2E)
6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. (2020). Thread: Circuits. Distill. [https://distill.pub/2020/circuits](https://distill.pub/2020/circuits). [↗](https://distill.pub/2020/circuits)
8. Droppo, J. and Elibol, O. (2021). Scaling laws for acoustic models. [find](https://scholar.google.com/scholar?q=Droppo%2C+J%2E+and+Elibol%2C+O%2E+%282021%29%2E+Scaling+laws+for+acoustic+models%2E)
9. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., a… [find](https://scholar.google.com/scholar?q=Elhage%2C+N%2E%2C+Nanda%2C+N%2E%2C+Olsson%2C+C%2E%2C+Henighan%2C+T%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Conerly%2C+T%2E%2C+DasSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Hernandez%2C+D%2E%2C+Jones%2C+A%2E%2C+K)
10. Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., A… [find](https://scholar.google.com/scholar?q=Ganguli%2C+D%2E%2C+Hernandez%2C+D%2E%2C+Lovitt%2C+L%2E%2C+DasSarma%2C+N%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Kernion%2C+J%2E%2C+Mann%2C+B%2E%2C+Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Conerly%2C+T%2E%2C+Drain%2C+D%2E%2C+Elhage%2C+N%2E%2C+Showk%2C+S%2E+E%2E%2C+Fort%2C)
11. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2021). The pile: An 800gb dataset of diverse text for language modeling. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+Presser%2C+S%2E%2C+and+Leahy%2C+C%2E+%282021%29%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
12. Geiger, M., Spigler, S., d’Ascoli, S., Sagun, L., Baity-Jesi, M., Biroli, G., and Wyart, M. (2019). Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):012115. [find](https://scholar.google.com/scholar?q=Geiger%2C+M%2E%2C+Spigler%2C+S%2E%2C+d%E2%80%99Ascoli%2C+S%2E%2C+Sagun%2C+L%2E%2C+Baity%2DJesi%2C+M%2E%2C+Biroli%2C+G%2E%2C+and+Wyart%2C+M%2E+%282019%29%2E+Jamming+transition+as+a+paradigm+to+understand+the+loss+landscape+of+deep+neural+networks%2E+Physical+)
13. Goh, G., Nick, C., Chelsea, V., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. (2021). Multimodal neurons in artificial neural networks. Distill. [https://distill.pub/2021/multimodal-neurons](https://distill.pub/2021/multimodal-neurons). [↗](https://distill.pub/2021/multimodal-neurons)
14. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. (2020). Scaling laws for autoregressive generative modeling. [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
15. Hernandez, D. and Brown, T. B. (2020). Measuring the algorithmic efficiency of neural networks. CoRR, abs/2005.04305. — [arXiv:2005.04305](https://arxiv.org/abs/2005.04305) [find](https://scholar.google.com/scholar?q=Hernandez%2C+D%2E+and+Brown%2C+T%2E+B%2E+%282020%29%2E+Measuring+the+algorithmic+efficiency+of+neural+networks%2E+CoRR%2C+abs%2F2005%2E04305%2E+%E2%80%94+%5BarXiv%3A2005%2E04305%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2005%2E04305%29)
16. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). Scaling laws for transfer. arXiv preprint arXiv:2102.01293. [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)
17. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
18. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. (2… [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Casas%2C+D%2E+d%2E+L%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+Hennigan%2C+T%2E%2C+Noland%2C+E%2E%2C+Millican%2C+K%2E%2C+Driessche%2C+G%2E+v%2E+d%2E%2C+)