---
title: "The Quantization Model of Neural Scaling — Michaud 2023"
description: "The paper tackles a puzzle about scaling. Average loss falls smoothly as a power law when models and datasets grow, yet specific abilities often seem to…"
authors: ["Eric J. Michaud", "Ziming Liu", "Uzay Girit", "Max Tegmark"]
year: 2023
venue: "arXiv:2303.13506"
source: "https://arxiv.org/abs/2303.13506"
doi: "10.48550/arxiv.2303.13506"
arxiv: "2303.13506"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / The Quantization Model of Neural Scaling

# The Quantization Model of Neural Scaling

**Authors:** Eric J. Michaud, Ziming Liu, Uzay Girit, [Max Tegmark](https://md.research.elamur.ai/free/en/authors/max-tegmark/) · **Year:** 2023 · **Venue:** arXiv:2303.13506 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.13506) · [PDF](https://arxiv.org/pdf/2303.13506) · [DOI 10.48550/arxiv.2303.13506](https://doi.org/10.48550/arxiv.2303.13506) · [arXiv 2303.13506](https://arxiv.org/abs/2303.13506) · [OpenAlex](https://openalex.org/W4360892090)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/michaud-2023-quantization-model-neural-scaling/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper tackles a puzzle about scaling. Average loss falls smoothly as a power law when models and datasets grow, yet specific abilities often seem to switch on abruptly. To reconcile the two, the authors propose the Quantization Hypothesis. It holds that prediction tasks rely on a countable set of discrete skills or knowledge units, called quanta. Each quantum is either acquired or missing, and models pick them up in order of how often they help on training data. If those usage frequencies follow a Zipf-like power law, the theory yields power-law loss curves in parameters and data. The smooth aggregate trend is then an average of many small, sudden jumps. The authors test this on synthetic toy tasks and then examine how scaling curves break down for real language models. They also introduce a gradient-clustering method, QDG, which automatically groups next-token predictions into coherent skills. The frequencies of the discovered clusters roughly match the power law implied by observed LLM scaling exponents, though the authors call this measurement imprecise. The work matters for forecasting new capabilities and for mechanistic interpretability.

## Key points
- **Core model:** the k-th quantum is useful on a fraction of samples proportional to k^−(α+1). A model that has learned n quanta has excess loss (L_n − L_∞) scaling as n^−α.
- **Parameter and data scaling:**
  - If each quantum needs about C parameters, the parameter exponent α_N equals α.
  - If each quantum needs τ training examples, the multi-epoch data exponent is α_D = α/(α+1).
  - Together these predict α_D = α_N/(α_N + 1).
- **Monogenic vs. polygenic samples:** the basic derivation assumes each sample depends on at most one quantum. The authors use this distinction to explain why some tasks improve gradually while others show breakthroughs.
- **Discovered skills:** QDG, applied to text from The Pile, recovers clusters such as incrementing numerical sequences (Cluster 50) and newline prediction in line-length-limited text (Cluster 100).
- **Limitations and availability:** the authors acknowledge that QDG is neither principled nor scalable. They also note that the quanta are modeled as independent, ignoring possible dependencies between skills. The paper appeared at NeurIPS 2023, and code is at github.com/ejmichaud/quantization-model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{michaud2023quantization,
  title = {{The Quantization Model of Neural Scaling}},
  author = {Michaud, Eric J. and Liu, Ziming and Girit, Uzay and Tegmark, Max},
  year = {2023},
  doi = {10.48550/arxiv.2303.13506},
  eprint = {2303.13506},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.13506}
}
```

- [← previous Prediction of Arabic Legal Rulings using Large Language Models](https://md.research.elamur.ai/free/en/papers/ammar-2023-prediction-arabic-legal-rulings-using/)
- [next → Are Large Language Models Robust Coreference Resolvers?](https://md.research.elamur.ai/free/en/papers/le-2023-large-language-models-robust-coreference/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

36 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. “Deep learning scaling is predictable, empirically”. In: arXiv preprint arXiv:1712.00409 (2017). Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, a… [find](https://scholar.google.com/scholar?q=Joel+Hestness%2C+Sharan+Narang%2C+Newsha+Ardalani%2C+Gregory+Diamos%2C+Heewoo+Jun%2C+Hassan+Kianinejad%2C+Md+Patwary%2C+Mostofa+Ali%2C+Yang+Yang%2C+and+Yanqi+Zhou%2E+%E2%80%9CDeep+learning+scaling+is+predictable%2C+empirically%E2%80%9D%2E+I) [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
2. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. “Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small”. In: arXiv preprint arXiv:2211.00593 (2022). [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+%E2%80%9CInterpretability+in+the+Wild%3A+a+Circuit+for+Indirect+Object+Identification+in+GPT%2D2+small%E2%80%9D%2E+In%3A+arXiv+preprint+ar)
3. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
4. Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. “Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla”. In: arXiv preprint arXiv:2307.09458 (2023). [find](https://scholar.google.com/scholar?q=Tom+Lieberum%2C+Matthew+Rahtz%2C+J%C3%A1nos+Kram%C3%A1r%2C+Geoffrey+Irving%2C+Rohin+Shah%2C+and+Vladimir+Mikulik%2E+%E2%80%9CDoes+circuit+analysis+interpretability+scale%3F+evidence+from+multiple+choice+capabilities+in+chinchilla%E2%80%9D%2E+) [our card](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/)
5. Stephen Casper, Shlomi Hod, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. “Graphical clusterability and local specialization in deep neural networks”. In: ICLR 2022 Workshop on PAIR {\\textasciicircum} 2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structur… [find](https://scholar.google.com/scholar?q=Stephen+Casper%2C+Shlomi+Hod%2C+Daniel+Filan%2C+Cody+Wild%2C+Andrew+Critch%2C+and+Stuart+Russell%2E+%E2%80%9CGraphical+clusterability+and+local+specialization+in+deep+neural+networks%E2%80%9D%2E+In%3A+ICLR+2022+Workshop+on+PAIR+%7B%5C%5Ct)
6. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah… [find](https://scholar.google.com/scholar?q=Trenton+Bricken%2C+Adly+Templeton%2C+Joshua+Batson%2C+Brian+Chen%2C+Adam+Jermyn%2C+Tom+Conerly%2C+Nick+Turner%2C+Cem+Anil%2C+Carson+Denison%2C+Amanda+Askell%2C+Robert+Lasenby%2C+Yifan+Wu%2C+Shauna+Kravec%2C+Nicholas+Schiefer%2C+)
7. Jonathan Frankle and Michael Carbin. “The lottery ticket hypothesis: Finding sparse, trainable neural networks”. In: arXiv preprint arXiv:1803.03635 (2018). [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+%E2%80%9CThe+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A1803%2E03635+%282018%29%2E)
8. Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. “Discovering Knowledge-Critical Subnetworks in Pretrained Language Models”. In: arXiv preprint arXiv:2310.03084 (2023). [find](https://scholar.google.com/scholar?q=Deniz+Bayazit%2C+Negar+Foroutan%2C+Zeming+Chen%2C+Gail+Weiss%2C+and+Antoine+Bosselut%2E+%E2%80%9CDiscovering+Knowledge%2DCritical+Subnetworks+in+Pretrained+Language+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2310%2E03084+%282023%29%2E)
9. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent Learning: Do different neural networks learn the same representations? arXiv:1511.07543 [cs]. Feb. 2016. URL : http://arxiv.org/abs/1511.07543 (visited on 08/19/2022). — [arXiv:1511.07543](https://arxiv.org/abs/1511.07543) [↗](http://arxiv.org/abs/1511.07543)
10. Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth. arXiv:2010.15327 [cs]. Apr. 2021. URL : http://arxiv.org/abs/2010.15327. — [arXiv:2010.15327](https://arxiv.org/abs/2010.15327) [↗](http://arxiv.org/abs/2010.15327)
11. Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta Neurons: Mining the Common Units in a Model Zoo. arXiv:2306.09346 [cs]. June 2023. DOI : 10.48550/arXiv. 2306.09346. URL : http://arxiv.org/abs/2306.09346. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) [↗](http://arxiv.org/abs/2306.09346)
12. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amode… [find](https://scholar.google.com/scholar?q=Catherine+Olsson%2C+Nelson+Elhage%2C+Neel+Nanda%2C+Nicholas+Joseph%2C+Nova+DasSarma%2C+Tom+Henighan%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
13. Gwern Branwen. The scaling hypothesis. 2021. URL : [https://gwern.net/scaling](https://gwern.net/scaling) -hypothesis. [↗](https://gwern.net/scaling)
14. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. “Grokking: Generalization beyond overfitting on small algorithmic datasets”. In: arXiv preprint arXiv:2201.02177 (2022). [find](https://scholar.google.com/scholar?q=Alethea+Power%2C+Yuri+Burda%2C+Harri+Edwards%2C+Igor+Babuschkin%2C+and+Vedant+Misra%2E+%E2%80%9CGrokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2201%2E02177+%282022%29%2E)
15. Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. “Hidden progress in deep learning: Sgd learns parities near the computational limit”. In: arXiv preprint arXiv:2207.08799 (2022). [find](https://scholar.google.com/scholar?q=Boaz+Barak%2C+Benjamin+L+Edelman%2C+Surbhi+Goel%2C+Sham+Kakade%2C+Eran+Malach%2C+and+Cyril+Zhang%2E+%E2%80%9CHidden+progress+in+deep+learning%3A+Sgd+learns+parities+near+the+computational+limit%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2)
16. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. “Pythia: A suite for analyzing large language models across training and scaling”. In: arXiv preprint arXiv:2304.01373 (2… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+%E2%80%9CPythia%3A+A+suite+for+)
17. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. “The pile: An 800gb dataset of diverse text for language modeling”. In: arXiv preprint arXiv:2101.00027 (2020). [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+%E2%80%9CThe+pile%3A+An+800gb+dataset+of+diverse+text+for+language)
18. Utkarsh Sharma and Jared Kaplan. “Scaling Laws from the Data Manifold Dimension”. In: Journal of Machine Learning Research 23.9 (2022), pp. 1–34. URL : [http://jmlr.org/](http://jmlr.org/) papers/v23/20-1111.html. [↗](http://jmlr.org/)
19. Eric J. Michaud, Ziming Liu, and Max Tegmark. “Precision Machine Learning”. In: Entropy 25.1 (2023). ISSN : 1099-4300. DOI : 10.3390/e25010175. URL : [https://www.mdpi.com/](https://www.mdpi.com/) 1099-4300/25/1/175. — [doi:10.3390/e25010175](https://doi.org/10.3390/e25010175) [↗](https://www.mdpi.com/)
20. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. “Explaining neural scaling laws”. In: arXiv preprint arXiv:2102.06701 (2021). — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) [find](https://scholar.google.com/scholar?q=Yasaman+Bahri%2C+Ethan+Dyer%2C+Jared+Kaplan%2C+Jaehoon+Lee%2C+and+Utkarsh+Sharma%2E+%E2%80%9CExplaining+neural+scaling+laws%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2102%2E06701+%282021%29%2E+%E2%80%94+%5BarXiv%3A2102%2E06701%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E)
21. Alexander Maloney, Daniel A Roberts, and James Sully. “A Solvable Model of Neural Scaling Laws”. In: arXiv preprint arXiv:2210.16859 (2022). — [arXiv:2210.16859](https://arxiv.org/abs/2210.16859) [find](https://scholar.google.com/scholar?q=Alexander+Maloney%2C+Daniel+A+Roberts%2C+and+James+Sully%2E+%E2%80%9CA+Solvable+Model+of+Neural+Scaling+Laws%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2210%2E16859+%282022%29%2E+%E2%80%94+%5BarXiv%3A2210%2E16859%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E16859%29)
22. Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. “Spectrum dependent learning curves in kernel regression and wide neural networks”. In: International Conference on Machine Learning. PMLR. 2020, pp. 1024–1034. [find](https://scholar.google.com/scholar?q=Blake+Bordelon%2C+Abdulkadir+Canatar%2C+and+Cengiz+Pehlevan%2E+%E2%80%9CSpectrum+dependent+learning+curves+in+kernel+regression+and+wide+neural+networks%E2%80%9D%2E+In%3A+International+Conference+on+Machine+Learning%2E+PMLR%2E+202)
23. Marcus Hutter. “Learning curve theory”. In: arXiv preprint arXiv:2102.04074 (2021). — [arXiv:2102.04074](https://arxiv.org/abs/2102.04074) [find](https://scholar.google.com/scholar?q=Marcus+Hutter%2E+%E2%80%9CLearning+curve+theory%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2102%2E04074+%282021%29%2E+%E2%80%94+%5BarXiv%3A2102%2E04074%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E04074%29)
24. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models”. In: arXiv preprint arXiv:2206.04615 (… [find](https://scholar.google.com/scholar?q=Aarohi+Srivastava%2C+Abhinav+Rastogi%2C+Abhishek+Rao%2C+Abu+Awal+Md+Shoeb%2C+Abubakar+Abid%2C+Adam+Fisch%2C+Adam+R+Brown%2C+Adam+Santoro%2C+Aditya+Gupta%2C+Adri%C3%A0+Garriga%2DAlonso%2C+et+al%2E+%E2%80%9CBeyond+the+imitation+game%3A+Quant) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
25. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. “Are Emergent Abilities of Large Language Models a Mirage?” In: arXiv preprint arXiv:2304.15004 (2023). [find](https://scholar.google.com/scholar?q=Rylan+Schaeffer%2C+Brando+Miranda%2C+and+Sanmi+Koyejo%2E+%E2%80%9CAre+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2304%2E15004+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/)
26. Sanjeev Arora and Anirudh Goyal. “A theory for emergence of complex skills in language models”. In: arXiv preprint arXiv:2307.15936 (2023). — [arXiv:2307.15936](https://arxiv.org/abs/2307.15936) [find](https://scholar.google.com/scholar?q=Sanjeev+Arora+and+Anirudh+Goyal%2E+%E2%80%9CA+theory+for+emergence+of+complex+skills+in+language+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2307%2E15936+%282023%29%2E+%E2%80%94+%5BarXiv%3A2307%2E15936%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E15936%29)
27. Lorenza Saitta, Attilio Giordana, and Antoine Cornuejols. Phase transitions in machine learning. Cambridge University Press, 2011. [find](https://scholar.google.com/scholar?q=Lorenza+Saitta%2C+Attilio+Giordana%2C+and+Antoine+Cornuejols%2E+Phase+transitions+in+machine+learning%2E+Cambridge+University+Press%2C+2011%2E)
28. James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. “On the stepwise nature of self-supervised learning”. In: arXiv preprint arXiv:2303.15438 (2023). — [arXiv:2303.15438](https://arxiv.org/abs/2303.15438) [find](https://scholar.google.com/scholar?q=James+B+Simon%2C+Maksis+Knutins%2C+Liu+Ziyin%2C+Daniel+Geisz%2C+Abraham+J+Fetterman%2C+and+Joshua+Albrecht%2E+%E2%80%9COn+the+stepwise+nature+of+self%2Dsupervised+learning%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2303%2E15438+%282023%29%2E+%E2%80%94+%5Ba)
29. Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. “Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models”. In: arXiv preprint arXiv:2307.14430 (2023). — [arXiv:2307.14430](https://arxiv.org/abs/2307.14430) [find](https://scholar.google.com/scholar?q=Mayee+F+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+R%C3%A9%2E+%E2%80%9CSkill%2Dit%21+A+Data%2DDriven+Skills+Framework+for+Understanding+and+Training+Language+Models%E2%80%9D%2E+In%3A+arXiv)
30. Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. “Data distributional properties drive emergent in-context learning in transformers”. In: Advances in Neural Information Processing Systems 35 (2022), pp. 18878–18891. [find](https://scholar.google.com/scholar?q=Stephanie+Chan%2C+Adam+Santoro%2C+Andrew+Lampinen%2C+Jane+Wang%2C+Aaditya+Singh%2C+Pierre+Richemond%2C+James+McClelland%2C+and+Felix+Hill%2E+%E2%80%9CData+distributional+properties+drive+emergent+in%2Dcontext+learning+in+trans)
31. Marvin Minsky. Society of mind. Simon and Schuster, 1988. [find](https://scholar.google.com/scholar?q=Marvin+Minsky%2E+Society+of+mind%2E+Simon+and+Schuster%2C+1988%2E)
32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12 (201… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
33. Ronen Eldan and Yuanzhi Li. “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?” In: arXiv preprint arXiv:2305.07759 (2023). [find](https://scholar.google.com/scholar?q=Ronen+Eldan+and+Yuanzhi+Li%2E+%E2%80%9CTinyStories%3A+How+Small+Can+Language+Models+Be+and+Still+Speak+Coherent+English%3F%E2%80%9D+In%3A+arXiv+preprint+arXiv%3A2305%2E07759+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/eldan-2023-tinystories-small-language-models-still/)
34. Pablo Villalobos. Scaling Laws Literature Review. [https://epochai.org/blog/scaling-laws-literature-review](https://epochai.org/blog/scaling-laws-literature-review). Accessed: 2023-01-31. 2023. [↗](https://epochai.org/blog/scaling-laws-literature-review)
35. Jasha Droppo and Oguz Elibol. “Scaling laws for acoustic models”. In: arXiv preprint arXiv:2106.09488 (2021). — [arXiv:2106.09488](https://arxiv.org/abs/2106.09488) [find](https://scholar.google.com/scholar?q=Jasha+Droppo+and+Oguz+Elibol%2E+%E2%80%9CScaling+laws+for+acoustic+models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2106%2E09488+%282021%29%2E+%E2%80%94+%5BarXiv%3A2106%2E09488%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E09488%29)
36. Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. “Understanding Scaling Laws for Recommendation Models”. In: arXiv preprint arXiv:2208.08489 (2022). — [arXiv:2208.08489](https://arxiv.org/abs/2208.08489) [find](https://scholar.google.com/scholar?q=Newsha+Ardalani%2C+Carole%2DJean+Wu%2C+Zeliang+Chen%2C+Bhargav+Bhushanam%2C+and+Adnan+Aziz%2E+%E2%80%9CUnderstanding+Scaling+Laws+for+Recommendation+Models%E2%80%9D%2E+In%3A+arXiv+preprint+arXiv%3A2208%2E08489+%282022%29%2E+%E2%80%94+%5BarXiv%3A2208%2E0848)