---
title: "Will we run out of data? Limits of LLM scaling based on"
description: "Scaling laws push LLM developers toward ever-larger training sets, which raises the question of… 118 references, cited by 119 papers here."
authors: ["Pablo Villalobos", "Anson Ho", "Jaime Sevilla", "Tamay Besiroglu", "Lennart Heim", "Marius Hobbhahn"]
year: 2022
venue: "arXiv:2211.04325"
source: "https://arxiv.org/abs/2211.04325"
doi: "10.48550/arxiv.2211.04325"
arxiv: "2211.04325"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Will we run out of data? Limits of LLM scaling based on human-generated data

# Will we run out of data? Limits of LLM scaling based on human-generated data

**Authors:** Pablo Villalobos, [Anson Ho](https://md.research.elamur.ai/free/en/authors/anson-ho/), Jaime Sevilla, [Tamay Besiroglu](https://md.research.elamur.ai/free/en/authors/tamay-besiroglu/), Lennart Heim and Marius Hobbhahn · **Year:** 2022 · **Venue:** arXiv:2211.04325 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.04325) · [PDF](https://arxiv.org/pdf/2211.04325) · [DOI 10.48550/arxiv.2211.04325](https://doi.org/10.48550/arxiv.2211.04325) · [arXiv 2211.04325](https://arxiv.org/abs/2211.04325) · [OpenAlex](https://openalex.org/W4308672130)

**Citations:** 79 citations (OpenAlex, 2026-10-14) · [119](https://md.research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Scaling laws push LLM developers toward ever-larger training sets, which raises the question of whether public human-written text will eventually run out. The authors, from Epoch, address this with a quantitative model built from two projections. The first extrapolates dataset sizes from recent notable LLMs. The second estimates the stock of public human text. To get that stock, they use Google's index as a stand-in for the size of the indexed web. They then multiply by the average plain text per page, measured on Common Crawl, convert bytes to tokens, and apply an annual growth rate. They also adjust for text quality and for repeating data over several epochs. Uncertainty is modelled with log-normal distributions and Monte Carlo simulation. The two curves cross somewhere between 2026 and 2032, most likely in 2028. If frontier models are overtrained, this happens a year or two earlier. The authors do not expect progress to stop there. They point to synthetic data, transfer from domains with plenty of data, and gains in data efficiency as ways forward. The paper treats data, not just compute, as a real limit on scaling.

## Key points
- The effective stock of indexed-web text is about 4e14 tokens. Training on all of it would need about 5e28 FLOP for a model that is not overtrained. With 5x overtraining, the stock is used up sooner.
- Raw stock estimates in tokens: Common Crawl about 130T, indexed web about 510T (95% CI 130T–2100T), whole web about 3100T. Images and video add roughly 300T and 1350T text-token equivalents.
- Model inputs: Google's index holds about 250B pages (CI 100B–1200B). A page has about 7000 bytes of plain text on average, up from 6100 in 2013 to 8200 in 2021. One token is about 4 bytes under the cl100k_base tokenizer. Web growth is assumed to be 0–10% per year.
- Data-selection methods such as pruning, domain mixing and curriculum learning are judged to give only small gains. One cited result found pruning helped less than tripling the dataset.
- The code is available at epochai.org/code/data-stock. The impact statement raises privacy risks in training on social-media and messaging data, and argues that data creators should be paid.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [MIT Computer Science and Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/mit-computer-science-and-artificial-intelligence-laboratory/) — Research institute, Cambridge, United States · 76 works · 32,207 citations
- [University of Tübingen](https://md.research.elamur.ai/free/en/universities/university-of-tubingen/) — University, Tübingen, Germany · 78 works · 41,280 citations
- [University of Aberdeen](https://md.research.elamur.ai/free/en/universities/university-of-aberdeen/) — University, Aberdeen, United Kingdom · 23 works · 1,583 citations
- [Centre for the Governance of AI](https://md.research.elamur.ai/free/en/ngos/centre-for-the-governance-of-ai/) — Nonprofit, Oxford, United Kingdom · 2 works · 302 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim and Marius Hobbhahn. 2022. Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv:2211.04325.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{villalobos2022will,
  title = {{Will we run out of data? Limits of LLM scaling based on human-generated data}},
  author = {Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim and Marius Hobbhahn},
  year = {2022},
  doi = {10.48550/arxiv.2211.04325},
  eprint = {2211.04325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.04325}
}
```

- [← previous Position Information in Transformers: An Overview](https://md.research.elamur.ai/free/en/papers/dufter-2022-position-information-transformers-overview/)
- [next → NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks](https://md.research.elamur.ai/free/en/papers/mishra-2022-numglue-suite-fundamental-yet-challenging/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

118 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training, 2021. URL https://arxiv.org/abs/2110.02095. — [arXiv:2110.02095](https://arxiv.org/abs/2110.02095) [↗](https://arxiv.org/abs/2110.02095)
2. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. et al. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Technical report, OpenAI, 2023. URL [https://arxiv.org/](https://arxiv.org/) abs/2303.08774. [↗](https://arxiv.org/)
3. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. [Weak-to-strong generalization: Eliciting strong capabilities with weak supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/), 2023.
4. Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. [Scaling laws for generative mixed-modal language models](https://md.research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/), 2023. URL https://arxiv.org/abs/2301.03728. [↗](https://arxiv.org/abs/2301.03728)
5. Campos, D. Curriculum learning for language modeling. CoRR, abs/2108.02170, 2021. URL https://arxiv.org/abs/2108.02170. — [arXiv:2108.02170](https://arxiv.org/abs/2108.02170) [↗](https://arxiv.org/abs/2108.02170)
6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., and Gehrmann, S. et al. [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/), 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2204.02311. [↗](https://arxiv.org/)
7. Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad, 2023. [find](https://scholar.google.com/scholar?q=Alemohammad%2C+S%2E%2C+Casco%2DRodriguez%2C+J%2E%2C+Luzi%2C+L%2E%2C+Humayun%2C+A%2E+I%2E%2C+Babaei%2C+H%2E%2C+LeJeune%2C+D%2E%2C+Siahkoohi%2C+A%2E%2C+and+Baraniuk%2C+R%2E+G%2E+Self%2Dconsuming+generative+models+go+mad%2C+2023%2E)
8. Cisco. Cisco Visual Networking Index: Forecast and Methodology, 2016–2021, 2017. [find](https://scholar.google.com/scholar?q=Cisco%2E+Cisco+Visual+Networking+Index%3A+Forecast+and+Methodology%2C+2016%E2%80%932021%2C+2017%2E)
9. Alpert, J. and Hajaj, N. We knew the web was big... [https://googleblog.blogspot.com/2008/07/we](https://googleblog.blogspot.com/2008/07/we) -knew-web-was-big.html, 2008. [Accessed 22-04-2024]. [↗](https://googleblog.blogspot.com/2008/07/we)
10. Coffman, K. and Odlyzko, A. The size and growth rate of the internet, 1998. URL https://firstmonday. [find](https://scholar.google.com/scholar?q=Coffman%2C+K%2E+and+Odlyzko%2C+A%2E+The+size+and+growth+rate+of+the+internet%2C+1998%2E+URL+https%3A%2F%2Ffirstmonday%2E)
11. Epoch. Key trends and figures in machine learning, 2023. URL [https://epochai.org/trends](https://epochai.org/trends). Accessed: 2024-01-27. org/ojs/index.php/fm/article/downloa d/620/541?inline=1. [↗](https://epochai.org/trends)
12. Common Crawl. Common crawl, 2024. URL [https://commoncrawl.org/](https://commoncrawl.org/). [↗](https://commoncrawl.org/)
13. Cottier, B. Trends in the dollar training cost of machine learning systems, 2023. URL [https://epochai.org/blog/trends-in-the-dollar-train](https://epochai.org/blog/trends-in-the-dollar-train) ing-cost-of-machine-learning-systems. Accessed: 2024-02-01. [↗](https://epochai.org/blog/trends-in-the-dollar-train)
14. Erdil, E. and Besiroglu, T. Algorithmic progress in computer vision, 2023. URL https://arxiv.org/abs/22 12.05153. [↗](https://arxiv.org/abs/22)
15. Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y. Scaling laws of synthetic images for model training ... for now, 2023. [find](https://scholar.google.com/scholar?q=Fan%2C+L%2E%2C+Chen%2C+K%2E%2C+Krishnan%2C+D%2E%2C+Katabi%2C+D%2E%2C+Isola%2C+P%2E%2C+and+Tian%2C+Y%2E+Scaling+laws+of+synthetic+images+for+model+training+%2E%2E%2E+for+now%2C+2023%2E)
16. Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. [Language modeling is compression](https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/), 2023. URL https://arxiv.org/abs/2309.10668. [↗](https://arxiv.org/abs/2309.10668)
17. Fetterman, A. J., Kitanidis, E., Albrecht, J., Polizzi, Z., Fo-gelman, B., Knutins, M., Wróblewski, B., Simon, J. B., and Qiu, K. Tune as you scale: Hyperparameter optimization for compute efficient training, 2023. URL https://arxiv.org/abs/2306.08055. — [arXiv:2306.08055](https://arxiv.org/abs/2306.08055) [↗](https://arxiv.org/abs/2306.08055)
18. Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebrasgpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. URL https://arxiv.org/abs/2304.03208. — [arXiv:2304.03208](https://arxiv.org/abs/2304.03208) [↗](https://arxiv.org/abs/2304.03208)
19. Friel, N., McKeone, J. P., Oates, C. J., and Pettitt, A. N. Investigation of the widely applicable bayesian information criterion, 2016. URL https://arxiv.org/abs/ 1501.05447. [↗](https://arxiv.org/abs/)
20. Gao, L. An empirical exploration in quality filtering of text data, 2021. URL https://arxiv.org/abs/2109 .00698. [↗](https://arxiv.org/abs/2109)
21. Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe, J. A tale of tails: Model collapse as a change of scaling laws, 2024. [find](https://scholar.google.com/scholar?q=Dohmatob%2C+E%2E%2C+Feng%2C+Y%2E%2C+Yang%2C+P%2E%2C+Charton%2C+F%2E%2C+and+Kempe%2C+J%2E+A+tale+of+tails%3A+Model+collapse+as+a+change+of+scaling+laws%2C+2024%2E)
22. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL https://arxiv.org/abs/2101.00027. [↗](https://arxiv.org/abs/2101.00027)
23. Domo. Data Never Sleeps 10.0. [https://www.do](https://www.do) mo.com/data-never-sleeps, 2022. [Accessed 12-04-2024]. [↗](https://www.do)
24. Dorner, F. E. Measuring progress in deep reinforcement learning sample efficiency, 2021. URL https://ar xiv.org/abs/2102.04881. — [arXiv:2102.04881](https://arxiv.org/abs/2102.04881) [find](https://scholar.google.com/scholar?q=Dorner%2C+F%2E+E%2E+Measuring+progress+in+deep+reinforcement+learning+sample+efficiency%2C+2021%2E+URL+https%3A%2F%2Far+xiv%2Eorg%2Fabs%2F2102%2E04881%2E+%E2%80%94+%5BarXiv%3A2102%2E04881%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E04881%29)
25. GDELT. Visualizing Twitter’s Evolution 2012-2020 And How Tweeting Is Changing In The COVID-19 Era. [https://blog.gdeltproject.org/visualiz](https://blog.gdeltproject.org/visualiz) ing-twitters-evolution-2012-2020-and -how-tweeting-is-changing-in-the-cov id-19-era/, 2020. [Accessed 12-04-2024]. [↗](https://blog.gdeltproject.org/visualiz)
26. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/), 2021. URL https://arxiv.org/abs/2010.11929. [↗](https://arxiv.org/abs/2010.11929)
27. Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., and Koyejo, S. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024. [find](https://scholar.google.com/scholar?q=Gerstgrasser%2C+M%2E%2C+Schaeffer%2C+R%2E%2C+Dey%2C+A%2E%2C+Rafailov%2C+R%2E%2C+Sleight%2C+H%2E%2C+Hughes%2C+J%2E%2C+Korbak%2C+T%2E%2C+Agrawal%2C+R%2E%2C+Pai%2C+D%2E%2C+Gromov%2C+A%2E%2C+Roberts%2C+D%2E+A%2E%2C+Yang%2C+D%2E%2C+Donoho%2C+D%2E+L%2E%2C+and+Koyejo%2C+S%2E+Is+model+collapse)
28. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
29. Griffin, A. Chatgpt creators openai are generating 100 billion words per day, ceo says, 2024. URL [https://www.independent.co.uk/tech/chatgpt-o](https://www.independent.co.uk/tech/chatgpt-o) penai-words-sam-altman-b2494900.html. [↗](https://www.independent.co.uk/tech/chatgpt-o)
30. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. [find](https://scholar.google.com/scholar?q=Gunasekar%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Aneja%2C+J%2E%2C+Mendes%2C+C%2E+C%2E+T%2E%2C+Giorno%2C+A%2E+D%2E%2C+Gopi%2C+S%2E%2C+Javaheripi%2C+M%2E%2C+Kauffmann%2C+P%2E%2C+de+Rosa%2C+G%2E%2C+Saarikivi%2C+O%2E%2C+Salim%2C+A%2E%2C+Shah%2C+S%2E%2C+Behl%2C+H%2E+S%2E%2C+Wang%2C+X%2E%2C+Bubeck%2C+S%2E%2C+Eldan)
31. Epoch. Parameter, compute and data trends in machine learning, 2022. URL [https://epochai.org/da](https://epochai.org/da) ta/epochdb/visualization. Accessed: 2024-01-29. [↗](https://epochai.org/da)
32. Haluptzok, P., Bowers, M., and Kalai, A. T. Language models can teach themselves to program better, 2023. URL https://arxiv.org/abs/2207.14502. [↗](https://arxiv.org/abs/2207.14502)
33. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/), 2020. URL https://arxiv.org/abs/2001.0 8361. [↗](https://arxiv.org/abs/2001.0)
34. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and Mc-Candlish, S. [Scaling laws for autoregressive generative modeling](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/), 2020. URL…
35. Kemp, S. Digital 2023: Global Overview Report — DataReportal. [https://datareportal.com/reports/digital-2023-global-overview-report](https://datareportal.com/reports/digital-2023-global-overview-report), 2023a. [Accessed 16-04-2024]. [↗](https://datareportal.com/reports/digital-2023-global-overview-report)
36. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. [Scaling laws for transfer](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/), 2021. URL https://ar xiv.org/abs/2102.01293.
37. Kemp, S. Facebook Messenger Users, Stats, Data, Trends, and More — DataReportal. https://datareport al.com/essential-facebook-messenger-s tats, 2023b. [Accessed 15-04-2024]. [find](https://scholar.google.com/scholar?q=Kemp%2C+S%2E+Facebook+Messenger+Users%2C+Stats%2C+Data%2C+Trends%2C+and+More+%E2%80%94+DataReportal%2E+https%3A%2F%2Fdatareport+al%2Ecom%2Fessential%2Dfacebook%2Dmessenger%2Ds+tats%2C+2023b%2E+%5BAccessed+15%2D04%2D2024%5D%2E)
38. Ho, A., Erdil, E., and Besiroglu, T. Limits to the energy efficiency of cmos microprocessors. In 2023 IEEE International Conference on Rebooting Computing (ICRC), pp. 1–10. [find](https://scholar.google.com/scholar?q=Ho%2C+A%2E%2C+Erdil%2C+E%2E%2C+and+Besiroglu%2C+T%2E+Limits+to+the+energy+efficiency+of+cmos+microprocessors%2E+In+2023+IEEE+International+Conference+on+Rebooting+Computing+%28ICRC%29%2C+pp%2E+1%E2%80%9310%2E)
39. Kemp, S. Facebook Users, Stats, Data, Trends, and More — DataReportal. [https://datareportal.com/e](https://datareportal.com/e) ssential-facebook-stats, 2023c. [Accessed 12-04-2024]. [↗](https://datareportal.com/e)
40. Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., and Sevilla, J. Algorithmic progress in language models. 2024. URL [https://api.semanticscholar.org/Corp](https://api.semanticscholar.org/Corp) usID:268358466. [↗](https://api.semanticscholar.org/Corp)
41. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., and Perez, E. [Debating with more persuasive llms leads to more truthful answers](https://md.research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/), 2024.
42. Krebs, F., Lubascher, B., Moers, T., Schaap, P., and Spanakis, G. Social emotion mining techniques for facebook posts reaction prediction, 2017. URL https://arxiv.org/abs/1712.03249. — [arXiv:1712.03249](https://arxiv.org/abs/1712.03249) [↗](https://arxiv.org/abs/1712.03249)
43. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Tr… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
44. Lee, E. 2021 worldwide image capture forecast: 2020 – 2025, 2021. URL [https://riseaboveresearch.com/rar-reports/2021-worldwide-image](https://riseaboveresearch.com/rar-reports/2021-worldwide-image) -capture-forecast-2020-2025/. [↗](https://riseaboveresearch.com/rar-reports/2021-worldwide-image)
45. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better, 2022. URL https://arxiv.org/abs/2107.06499. [↗](https://arxiv.org/abs/2107.06499)
46. Howell, S. and Burtis, A. T. The continued problem of url decay: an updated analysis of health care management journal citations. Journal of the Medical Library Association : JMLA, 110:463 – 470, 2022. URL [https://api.se](https://api.se) manticscholar.org/CorpusID:257746935. [↗](https://api.se)
47. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463, 2023.
48. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/), 2022. URL https://arxiv.org/abs/2210.11610. [↗](https://arxiv.org/abs/2210.11610)
49. Liu, B., Bubeck, S., Eldan, R., Kulkarni, J., Li, Y., Nguyen, A., Ward, R., and Zhang, Y. Tinygsm: achieving ¿80 Institute, N. H. G. R. Genomic data science (fact sheet), 2024. URL [https://www.genome.gov/about](https://www.genome.gov/about) -genomics/fact-sheets/Genomic-Data-S cience. [↗](https://www.genome.gov/about)
50. Loan, F. A. and Shah, U. Y. The decay and persistence of web references. Digit. Libr. Perspect., 36:157–166, 2020. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:219404788. [↗](https://api.semanticscholar.org/)
51. Irving, G., Christiano, P., and Amodei, D. [Ai safety via debate](https://md.research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/), 2018.
52. Loan, F. A., Khan, A. M., Andrabi, S. A. A., Sozia, S. R., and Parray, U. Y. Giving life to dead: role of WayBack Machine in recovery of dead URLs. Data Technologies and Applications, 2023. URL [https://api.sema](https://api.sema) nticscholar.org/CorpusID:259585485. jerryspan. GitHub - jerryspan/FacebookR: Facebook Pos… [↗](https://api.sema)
53. Lyman, P. and Varian, H. R. How much information, 2003. URL [https://groups.ischool.berkeley.edu/archive/how-much-info-2003/](https://groups.ischool.berkeley.edu/archive/how-much-info-2003/). [↗](https://groups.ischool.berkeley.edu/archive/how-much-info-2003/)
54. Mahoney, M. Large Text Compression Benchmark, 2006/2024. URL [http://www.mattmahoney.net/dc/text.html](http://www.mattmahoney.net/dc/text.html). [Accessed 29-01-2024]. [↗](http://www.mattmahoney.net/dc/text.html)
55. Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of clip, 2022. URL https://arxiv.org/abs/2208.05516. — [arXiv:2208.05516](https://arxiv.org/abs/2208.05516) [↗](https://arxiv.org/abs/2208.05516)
56. Nostalgebraist. chinchilla’s wild implications, 2022. URL [https://www.lesswrong.com/posts/6Fpv](https://www.lesswrong.com/posts/6Fpv) ch8RR29qLEWNH/chinchilla-s-wild-impli cations. [↗](https://www.lesswrong.com/posts/6Fpv)
57. Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale, 2023. URL https://arxiv.org/abs/2309.04564. [↗](https://arxiv.org/abs/2309.04564)
58. OBDILCI. Indicators for the Presence of Languages in the Internet. [https://www.obdilci.org/projec](https://www.obdilci.org/projec) ts/main/, 2024. [Accessed 16-04-2024]. [↗](https://www.obdilci.org/projec)
59. Meta. Introducing Meta Llama 3: The most capable openly available LLM to date. [https://ai.meta.com/bl](https://ai.meta.com/bl) og/meta-llama-3/, 2024. [Accessed 24-04-2024]. [↗](https://ai.meta.com/bl)
60. Odlyzko, A. The growth rate and the nature of internet traffic. Transactions on Internet Research, pp. 39–42, 2016. [find](https://scholar.google.com/scholar?q=Odlyzko%2C+A%2E+The+growth+rate+and+the+nature+of+internet+traffic%2E+Transactions+on+Internet+Research%2C+pp%2E+39%E2%80%9342%2C+2016%2E)
61. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts, 2023. [find](https://scholar.google.com/scholar?q=Michael%2C+J%2E%2C+Mahdi%2C+S%2E%2C+Rein%2C+D%2E%2C+Petty%2C+J%2E%2C+Dirani%2C+J%2E%2C+Padmakumar%2C+V%2E%2C+and+Bowman%2C+S%2E+R%2E+Debate+helps+supervise+unreliable+experts%2C+2023%2E)
62. OECD. Gross domestic spending on R&D (indicator), 2022. URL [https://data.oecd.org/rd/gross-d](https://data.oecd.org/rd/gross-d) omestic-spending-on-r-d.htm. [↗](https://data.oecd.org/rd/gross-d)
63. Mitchell, M., Luccioni, A. S., Lambert, N., Gerchick, M., McMillan-Major, A., Ozoani, E., Rajani, N., Thrush, T., Jernite, Y., and Kiela, D. Measuring data, 2023. URL https://arxiv.org/abs/2212.05129. — [arXiv:2212.05129](https://arxiv.org/abs/2212.05129) [↗](https://arxiv.org/abs/2212.05129)
64. OpenAI. Gpt-4v(ision) system card. Technical report, OpenAI, 2023. Accessed: 2024-02-01. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4v%28ision%29+system+card%2E+Technical+report%2C+OpenAI%2C+2023%2E+Accessed%3A+2024%2D02%2D01%2E)
65. Mosaic AI. Introducing DBRX: A New State-of-the-Art Open LLM — Databricks — databricks.com. [https://www.databricks.com/blog/introduc](https://www.databricks.com/blog/introduc) ing-dbrx-new-state-art-open-llm, 2024. [Accessed 12-04-2024]. [↗](https://www.databricks.com/blog/introduc)
66. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, 2021. URL [https://openai.com/blog/](https://openai.com/blog/) gpt-3-apps/. [↗](https://openai.com/blog/)
67. OpenAI. tiktoken, 2024. URL [https://github.com/openai/tiktoken](https://github.com/openai/tiktoken). [↗](https://github.com/openai/tiktoken)
68. Mosseri, A. and Chudnovsky, S. Say :wave: to Messenger: Introducing New Messaging Features for Instagram. [https://about.fb.com/news/2020/09/new-m](https://about.fb.com/news/2020/09/new-m) essaging-features-for-instagram/, 2020. [Accessed 15-04-2024]. [↗](https://about.fb.com/news/2020/09/new-m)
69. OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. Solving rubik’s cube with a robot hand, 2019. [find](https://scholar.google.com/scholar?q=OpenAI%2C+Akkaya%2C+I%2E%2C+Andrychowicz%2C+M%2E%2C+Chociej%2C+M%2E%2C+Litwin%2C+M%2E%2C+McGrew%2C+B%2E%2C+Petron%2C+A%2E%2C+Paino%2C+A%2E%2C+Plappert%2C+M%2E%2C+Powell%2C+G%2E%2C+Ribas%2C+R%2E%2C+Schneider%2C+J%2E%2C+Tezak%2C+N%2E%2C+Tworek%2C+J%2E%2C+Welinder%2C+P%2E%2C+Weng%2C+L%2E%2C+Yua)
70. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models, 2023. URL https://arxiv.org/abs/2305.16264. [↗](https://arxiv.org/abs/2305.16264)
71. Ott, D. E. Reference hygiene and death on the internet– decay, rot, half-life, deterioration, and corruption. JSLS: Journal of the Society of Laparoscopic & Robotic Surgeons, 26(1), 2022. [find](https://scholar.google.com/scholar?q=Ott%2C+D%2E+E%2E+Reference+hygiene+and+death+on+the+internet%E2%80%93+decay%2C+rot%2C+half%2Dlife%2C+deterioration%2C+and+corruption%2E+JSLS%3A+Journal+of+the+Society+of+Laparoscopic+%26+Robotic+Surgeons%2C+26%281%29%2C+2022%2E)
72. Murray H., B. and Moore, A. Sizing the internet. Technical report, Cyveillance, 7 2000. URL [http://www.cs.t](http://www.cs.t) oronto.edu/˜leehyun/papers/Sizing_the _Internet.pdf. [↗](http://www.cs.t)
73. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language models to follow instruct… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
74. National Electrical Manufacturers Association (NEMA). Comments on innovative advanced transformers. RFI Response DE-FOA-0003021, U.S. Department of Energy, Washington, DC, May 2023. URL [https://www.ne](https://www.ne) ma.org/docs/default-source/advocacy-d ocument-library/nema-comments-on-inn ovative-advanced-transfo… [↗](https://www.ne)
75. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. URL https://arxiv.org/abs/2306 .01116. [↗](https://arxiv.org/abs/2306)
76. Pichai, S. and Hassabis, D. Introducing gemini: our largest and most capable ai model, 2023. URL https://bl og.google/technology/ai/google-gemin i-ai/. [find](https://scholar.google.com/scholar?q=Pichai%2C+S%2E+and+Hassabis%2C+D%2E+Introducing+gemini%3A+our+largest+and+most+capable+ai+model%2C+2023%2E+URL+https%3A%2F%2Fbl+og%2Egoogle%2Ftechnology%2Fai%2Fgoogle%2Dgemin+i%2Dai%2F%2E)
77. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/), 2022.
78. Scao, T. L., Wang, T., Hesslow, D., Saulnier, L., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., Raffel, C., Sanh, V., Shen, S., Sutawika, L., Tae, J., Yong, Z. X., Launay, J., and Beltagy, I. [What language model to train if you have one million gpu hours?](https://md.research.elamur.ai/free/en/papers/scao-2022-language-model-train-if-have/)…
79. Radicati. Email statistics report, 2020-2024. https://ww w.radicati.com/wp/wp-content/uploads /2020/01/Email_Statistics_Report,_20 20-2024_Executive_Summary.pdf. [Accessed 03-05-2024]. [find](https://scholar.google.com/scholar?q=Radicati%2E+Email+statistics+report%2C+2020%2D2024%2E+https%3A%2F%2Fww+w%2Eradicati%2Ecom%2Fwp%2Fwp%2Dcontent%2Fuploads+%2F2020%2F01%2FEmail%5FStatistics%5FReport%2C%5F20+20%2D2024%5FExecutive%5FSummary%2Epdf%2E+%5BAccessed+03%2D05%2D2024%5D%2E)
80. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67, 2019. URL https://arxiv.org/abs/1910.10683. [↗](https://arxiv.org/abs/1910.10683)
81. Schuhmann, C. and Sun, Q. Strategic game datasets for enhancing ai planning: An invitation for collaborative research. LAION Projects Team Blog, Oct 2023. URL [https://laion.ai/blog/strategic-gam](https://laion.ai/blog/strategic-gam) e-dataset/. Accessed: [insert date here]. [↗](https://laion.ai/blog/strategic-gam)
82. Real Time Statistics Project. Internet Live Stats. URL [https://www.internetlivestats.com/](https://www.internetlivestats.com/). [Accessed 02-02-2024]. [↗](https://www.internetlivestats.com/)
83. Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., and Villalobos, P. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN). [find](https://scholar.google.com/scholar?q=Sevilla%2C+J%2E%2C+Heim%2C+L%2E%2C+Ho%2C+A%2E%2C+Besiroglu%2C+T%2E%2C+Hobbhahn%2C+M%2E%2C+and+Villalobos%2C+P%2E+Compute+trends+across+three+eras+of+machine+learning%2E+In+2022+International+Joint+Conference+on+Neural+Networks+%28IJCNN%29%2E)
84. IEEE, 2022. doi: 10.1109/ ijcnn55064.2022.9891914. URL http://dx.doi.org/10.1109/IJCNN55064.2022.9891914. — [doi:10.1109/ijcnn55064.2022.9891914](https://doi.org/10.1109/ijcnn55064.2022.9891914) [↗](http://dx.doi.org/10.1109/IJCNN55064.2022.9891914)
85. Reinsel, D., Gantz, J., and Rydning, J. The digitization of the world from edge to core. Technical report, International Data Corporation, 11 2018. URL [https://www.seagate.com/files/www-c](https://www.seagate.com/files/www-c) ontent/our-story/trends/files/idc-sea gate-dataage-whitepaper.pdf. [↗](https://www.seagate.com/files/www-c)
86. Shepard, C. Google’s index size revealed: 400 billion docs (& changing), 2024. URL [https://zyppy.com/se](https://zyppy.com/se) o/google-index-size/. [↗](https://zyppy.com/se)
87. Ritchie, H. and Roser, M. Technology adoption. Our World in Data, 2017. URL [https://ourworldindata.org/technology-adoption](https://ourworldindata.org/technology-adoption). [↗](https://ourworldindata.org/technology-adoption)
88. Shilov, A. Nvidia to reportedly triple output of compute gpus in 2024: Up to 2 million h100s, Aug 2023. URL [https://www.tomshardware.com/news/nvid](https://www.tomshardware.com/news/nvid) ia-to-reportedly-triple-output-of-c ompute-gpus-in-2024-up-to-2-million -h100s. [↗](https://www.tomshardware.com/news/nvid)
89. Ritchie, H. and Roser, M. Number of people using the internet. Our World in Data, 2020. URL https://ou rworldindata.org/grapher/number-of-i nternet-users. [find](https://scholar.google.com/scholar?q=Ritchie%2C+H%2E+and+Roser%2C+M%2E+Number+of+people+using+the+internet%2E+Our+World+in+Data%2C+2020%2E+URL+https%3A%2F%2Fou+rworldindata%2Eorg%2Fgrapher%2Fnumber%2Dof%2Di+nternet%2Dusers%2E)
90. Roose, K. and Newton, C. Dario amodei on the paradoxes of a.i. safety and netflix’s ’deep fake love’. Podcast Transcript, July 2023. URL [https://www.nytimes.com/section/technology](https://www.nytimes.com/section/technology). Available at The New York Times. [↗](https://www.nytimes.com/section/technology)
91. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/), 2023. URL https://arxiv.org/abs/2305.17493. [↗](https://arxiv.org/abs/2305.17493)
92. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. [Mastering chess and shogi by self-play with a general reinforcement learning algorithm](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/), 2017. URL https://arxiv.org/abs/1712.0… [↗](https://arxiv.org/abs/1712.0…)
93. Rosenfeld, A., Sina, S., Sarne, D., Avidov, O., and Kraus, S. WhatsApp usage patterns and prediction of demographic characteristics without access to message content. Demographic Research, 39(22):647–670, 2018. doi: 10.4054/DemRes.2018.39.22. URL [https://www.demographic-research.org/volume](https://www.demographic-research.org/volume) s/vol39/… — [doi:10.4054/demres.2018.39.22](https://doi.org/10.4054/demres.2018.39.22) [↗](https://www.demographic-research.org/volume)
94. Similarweb. Top Websites. [https://www.similarw](https://www.similarw) eb.com/top-websites/, 2024. [Accessed 16-04-2024]. [↗](https://www.similarw)
95. Sardana, N. and Frankle, J. [Beyond chinchilla-optimal: Accounting for inference in language model scaling laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/), 2023. URL https://arxiv.org/abs/2401.0 0448. [↗](https://arxiv.org/abs/2401.0)
96. Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., Kumar, A., Alemi, A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G., Sedghi, H., Mordatch, I., Simpson, I., Gur, I., Snoek, J., Pennington, J., Satyanarayan… [find](https://scholar.google.com/scholar?q=Singh%2C+A%2E%2C+Co%2DReyes%2C+J%2E+D%2E%2C+Agarwal%2C+R%2E%2C+Anand%2C+A%2E%2C+Patil%2C+P%2E%2C+Garcia%2C+X%2E%2C+Liu%2C+P%2E+J%2E%2C+Harrison%2C+J%2E%2C+Lee%2C+J%2E%2C+Xu%2C+K%2E%2C+Parisi%2C+A%2E%2C+Kumar%2C+A%2E%2C+Alemi%2C+A%2E%2C+Rizkowsky%2C+A%2E%2C+Nova%2C+A%2E%2C+Adlam%2C+B%2E%2C+Bohnet%2C+B%2E%2C+)
97. Hron, J., Kenealy, K., Swersky, K., Mahajan, K., Culp, L., Xiao, L., Bileschi, M. L., Constant, N., Novak, R., Liu, R., Warkentin, T., Qian, Y., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel, N. Beyond human data: Scaling self-training for problem-solving with language models,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
98. Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T. [Solving olympiad geometry without human demonstrations](https://md.research.elamur.ai/free/en/papers/trinh-2024-solving-olympiad-geometry-without-human/). Nature, 625(7995):476–482, January 2024. ISSN 1476-4687. doi: 10.1038/s41586-023-06747-5. URL http://dx.doi.org/10.1038/s41586-023 -06747-5. [↗](http://dx.doi.org/10.1038/s41586-023)
99. Singh, M. WhatsApp is now delivering roughly 100 billion messages a day. [https://techcrunch.com/2](https://techcrunch.com/2) 020/10/29/whatsapp-is-now-deliverin g-roughly-100-billion-messages-a-day, 2020. [Accessed 15-04-2024]. [↗](https://techcrunch.com/2)
100. United Nations. World population prospects 2022, online edition, 2022. URL [https://population.un.org/wpp/](https://population.un.org/wpp/). [↗](https://population.un.org/wpp/)
101. Size, W. W. W. World wide web size, 2024. URL [https://www.worldwidewebsize.com/](https://www.worldwidewebsize.com/). [↗](https://www.worldwidewebsize.com/)
102. Stephens, Z. D., Lee, S., Faghri, F., Campbell, R. H., Zhai, C., Efron, M., Iyer, R. K., Schatz, M. C., Sinha, S., and Robinson, G. E. Big data: Astronomical or genomical? PLoS Biology, 13, 2015. URL [https://journals.plos.org/plosbiology/article?id=10](https://journals.plos.org/plosbiology/article?id=10). 1371/journal.pbio.1002195. [↗](https://journals.plos.org/plosbiology/article?id=10)
103. Taniguchi, M., Ueda, Y., Taniguchi, T., and Ohkuma, T. [A large-scale corpus of E-mail conversations with standard and two-level dialogue act annotations](https://md.research.elamur.ai/free/en/papers/taniguchi-2020-large-scale-corpus-e-mail/). In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 4969–4980, Barcelon…
104. Villalobos, P. and Ho, A. Trends in training dataset sizes. [https://epochai.org/blog/trends-in-t](https://epochai.org/blog/trends-in-t) raining-dataset-sizes, 2022. Accessed: 2022-09-27. [↗](https://epochai.org/blog/trends-in-t)
105. Wei, T., Zhao, L., Zhang, L., Zhu, B., Wang, L., Yang, H., Li, B., Cheng, C., Lü, W., Hu, R., Li, C., Yang, L., Luo, X., Wu, X., Liu, L., Cheng, W., Cheng, P., Zhang, J., Zhang, X., Lin, L., Wang, X., Ma, Y., Dong, C., Sun, Y., Chen, Y., Peng, Y., Liang, X., Yan, S., Fang, H., and Zhou, Y. Skywork:… — [Skywork: A More Open Bilingual Foundation Model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
106. White, K. Publications output: U.s. trends and international comparisons, 2019. URL [https://ncses.nsf.gov/pubs/nsb20206/](https://ncses.nsf.gov/pubs/nsb20206/). [↗](https://ncses.nsf.gov/pubs/nsb20206/)
107. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/), 2022. URL https://arxiv.org/abs/2211.09085. [↗](https://arxiv.org/abs/2211.09085)
108. Thorgren, E., Mohammadinodooshan, A., and Carlsson, N. Temporal dynamics of user engagement on instagram: A comparative analysis of album, photo, and video interactions. URL — [D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/) [https://api.semanticscholar.org/CorpusID:268241757](https://api.semanticscholar.org/CorpusID:268241757). Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. S. D4: Improving… [↗](https://api.semanticscholar.org/CorpusID:268241757)
109. Wikipedia. Wikipedia:Size of Wikipedia. [https://en.wikipedia.org/wiki/Wikipedia:Size_o](https://en.wikipedia.org/wiki/Wikipedia:Size_o) f_Wikipedia. [Accessed 22-04-2024]. [↗](https://en.wikipedia.org/wiki/Wikipedia:Size_o)
110. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre-training, 2023. URL https://arxiv.org/abs/ 2305.10429. [↗](https://arxiv.org/abs/)
111. Xue, F., Fu, Y., Zhou, W., Zheng, Z., and You, Y. To repeat or not to repeat: Insights from scaling llm under token-crisis, 2023. URL https://arxiv.org/abs/23 05.13230. [↗](https://arxiv.org/abs/23)
112. Together.ai. Redpajama-data-v2: An open dataset with 30 trillion tokens for training large language models, 2023. URL [https://www.together.ai/blog/red](https://www.together.ai/blog/red) pajama-data-v2. [↗](https://www.together.ai/blog/red)
113. XVERSE Technology Inc. Xverse-65b: A multilingual large language model, 2024. URL https://gith ub.com/xverse-ai/XVERSE-65B. Apache-2.0 License. [find](https://scholar.google.com/scholar?q=XVERSE+Technology+Inc%2E+Xverse%2D65b%3A+A+multilingual+large+language+model%2C+2024%2E+URL+https%3A%2F%2Fgith+ub%2Ecom%2Fxverse%2Dai%2FXVERSE%2D65B%2E+Apache%2D2%2E0+License%2E)
114. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., A. Theoretical growth model of the web Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+A%2E+Theoretical+growth+model+of+the+web+Yang%2C+G%2E%2C+Hu%2C+E%2E+J%2E%2C+Babuschkin%2C+I%2E%2C+Sidor%2C+S%2E)
115. We explain in more detail our theoretical model of data accumulation rates developed in Section 2.2.2 and check it on Reddit submission data. The model is explained in Equation 5. Yang, K., Swope, A. M., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R., and Anandkumar, A. Leandojo: Th… — [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/)
116. A purely exponential model cannot reproduce the decrease in the growth rate of Reddit submissions over time, while a purely sigmoidal model plateaus at zero growth. The exponential times sigmoid model is able to better capture the deceleration in submission size growth (see Figure 7). Ye, W., Liu, S… — [arXiv:2111.00210](https://arxiv.org/abs/2111.00210) [find](https://scholar.google.com/scholar?q=A+purely+exponential+model+cannot+reproduce+the+decrease+in+the+growth+rate+of+Reddit+submissions+over+time%2C+while+a+purely+sigmoidal+model+plateaus+at+zero+growth%2E+The+exponential+times+sigmoid+model)
117. In our actual model, the slowdown in population growth (which becomes subexponential) leads to additional deceleration, but the time period covered by the Reddit submission dataset seems too short for slowing population growth to be noticeable in the data. Yole Développement. Status of the CMOS Ima… [find](https://scholar.google.com/scholar?q=In+our+actual+model%2C+the+slowdown+in+population+growth+%28which+becomes+subexponential%29+leads+to+additional+deceleration%2C+but+the+time+period+covered+by+the+Reddit+submission+dataset+seems+too+short+for)
118. Projection Based on the Number of Internet Users Z y [find](https://scholar.google.com/scholar?q=Projection+Based+on+the+Number+of+Internet+Users+Z+y)

*Updated 2 November 2026*