---
title: "Language Modeling Is Compression — Deletang 2023"
description: "The work matters because it offers compression as a principled lens on generalization, scaling and in-context… 82 references, cited by 60 papers here."
authors: ["Gregoire Deletang", "Anian Ruoss", "Paul-Ambroise Duquenne", "Elliot Catt", "Tim Genewein", "Christopher Mattern", "Jordi Grau-Moya", "Li Kevin Wenliang", "Matthew Aitchison", "Laurent Orseau", "Marcus Hutter", "Joel Veness"]
year: 2023
venue: "arXiv:2309.10668"
source: "https://arxiv.org/abs/2309.10668"
doi: "10.48550/arxiv.2309.10668"
arxiv: "2309.10668"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Language Modeling Is Compression

# Language Modeling Is Compression

**Authors:** [Gregoire Deletang](https://md.research.elamur.ai/free/en/authors/gregoire-deletang/), [Anian Ruoss](https://md.research.elamur.ai/free/en/authors/anian-ruoss/), [Paul-Ambroise Duquenne](https://md.research.elamur.ai/free/en/authors/paul-ambroise-duquenne/), [Elliot Catt](https://md.research.elamur.ai/free/en/authors/elliot-catt/), [Tim Genewein](https://md.research.elamur.ai/free/en/authors/tim-genewein/), Christopher Mattern, [Jordi Grau-Moya](https://md.research.elamur.ai/free/en/authors/jordi-grau-moya/), [Li Kevin Wenliang](https://md.research.elamur.ai/free/en/authors/li-kevin-wenliang/), Matthew Aitchison, Laurent Orseau, [Marcus Hutter](https://md.research.elamur.ai/free/en/authors/marcus-hutter/) and [Joel Veness](https://md.research.elamur.ai/free/en/authors/joel-veness/) · **Year:** 2023 · **Venue:** arXiv:2309.10668 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.10668) · [PDF](https://arxiv.org/pdf/2309.10668) · [DOI 10.48550/arxiv.2309.10668](https://doi.org/10.48550/arxiv.2309.10668) · [arXiv 2309.10668](https://arxiv.org/abs/2309.10668) · [OpenAlex](https://openalex.org/W4386908184)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper revisits the old equivalence between prediction and lossless compression and uses it to evaluate large pretrained models. Any model that assigns probabilities to the next symbol can drive an arithmetic coder, so a lower log-loss means a shorter code. The authors, from Google DeepMind and Meta AI/Inria, test pretrained models in an offline setting. Parameters stay frozen and each data chunk is compressed using only the model's context window. They compare Chinchilla models (1B, 7B, 70B) and Transformers trained on enwik against standard compressors on text, image and audio data. Despite being trained mostly on text, the largest model beats domain-specific tools such as PNG and FLAC, if model size is ignored. Once the parameters are counted as part of the output, scaling has a limit: beyond some point, bigger models compress a given dataset worse. The authors also show that tokenization does not generally help compression. Finally, they run the equivalence in reverse to sample from gzip as a generative model. The work matters because it offers compression as a principled lens on generalization, scaling and in-context learning.

## Key points
- Chinchilla 70B compresses ImageNet patches to 43.4% of their raw size, compared with 58.5% for PNG. It compresses LibriSpeech samples to 16.4%, compared with 30.3% for FLAC.
- Inputs are split into 2048-byte chunks. Because the large models expose only their top-k next-token log-probabilities (k = 100), these are renormalized before being passed to the arithmetic coder.
- The authors propose an adjusted compression rate that includes model size. Under this measure, dataset size puts a hard ceiling on useful model size, so model scaling is not a cure-all.
- Tokenization acts as a form of pre-compression. It does not usually improve compression rates, but it lets more information fit into the context, which helps prediction.
- In generation experiments conditioned on 1948 bytes of enwik9 text, Chinchilla samples are coherent while gzip samples are mostly noise. Similar comparisons are shown for image and LibriSpeech audio data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

## Cite

Cite as: Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter and Joel Veness. 2023. Language Modeling Is Compression. arXiv:2309.10668.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{deletang2023language,
  title = {{Language Modeling Is Compression}},
  author = {Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter and Joel Veness},
  year = {2023},
  doi = {10.48550/arxiv.2309.10668},
  eprint = {2309.10668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.10668}
}
```

- [← previous Skywork: A More Open Bilingual Foundation Model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
- [next → Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-bridging-gap-survey-integrating-human/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

82 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Benjamin Lukas Cajus Barzen, Fedor Glazov, Jonas Geistert, and Thomas Sikora. Accelerated deep lossless image coding with unified paralleleized GPU coding architecture. In PCS, 2022. [find](https://scholar.google.com/scholar?q=Benjamin+Lukas+Cajus+Barzen%2C+Fedor+Glazov%2C+Jonas+Geistert%2C+and+Thomas+Sikora%2E+Accelerated+deep+lossless+image+coding+with+unified+paralleleized+GPU+coding+architecture%2E+In+PCS%2C+2022%2E)
2. Fabrice Bellard. Lossless data compression with neural networks. Technical report, Amarisoft, 2019. [find](https://scholar.google.com/scholar?q=Fabrice+Bellard%2E+Lossless+data+compression+with+neural+networks%2E+Technical+report%2C+Amarisoft%2C+2019%2E)
3. Fabrice Bellard. NNCP v2: Lossless data compression with transformer. Technical report, Amarisoft, 2021. [find](https://scholar.google.com/scholar?q=Fabrice+Bellard%2E+NNCP+v2%3A+Lossless+data+compression+with+transformer%2E+Technical+report%2C+Amarisoft%2C+2021%2E)
4. Léonard Blier and Yann Ollivier. The description length of deep learning models. In NeurIPS, 2018. [find](https://scholar.google.com/scholar?q=L%C3%A9onard+Blier+and+Yann+Ollivier%2E+The+description+length+of+deep+learning+models%2E+In+NeurIPS%2C+2018%2E)
5. Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv:2108.07258, 2021. [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
6. Thomas Boutell. PNG (portable network graphics) specification version 1.0. RFC, 1997. [find](https://scholar.google.com/scholar?q=Thomas+Boutell%2E+PNG+%28portable+network+graphics%29+specification+version+1%2E0%2E+RFC%2C+1997%2E)
7. Tom B. Brown, Benjamin Mannand Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. In NeurIPS, 2020. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv:2303.127… [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
9. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with RMT. arXiv:2304.11062, 2023. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yuri+Kuratov%2C+and+Mikhail+S%2E+Burtsev%2E+Scaling+transformer+to+1m+tokens+and+beyond+with+RMT%2E+arXiv%3A2304%2E11062%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E11062%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E11062%29)
10. Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv:1710.09282, 2017. — [arXiv:1710.09282](https://arxiv.org/abs/1710.09282) [find](https://scholar.google.com/scholar?q=Yu+Cheng%2C+Duo+Wang%2C+Pan+Zhou%2C+and+Tao+Zhang%2E+A+survey+of+model+compression+and+acceleration+for+deep+neural+networks%2E+arXiv%3A1710%2E09282%2C+2017%2E+%E2%80%94+%5BarXiv%3A1710%2E09282%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1710%2E09282%29)
11. John G. Cleary and Ian H. Witten. Data compression using adaptive coding and partial string matching. IEEE Trans. Commun., 1984. [find](https://scholar.google.com/scholar?q=John+G%2E+Cleary+and+Ian+H%2E+Witten%2E+Data+compression+using+adaptive+coding+and+partial+string+matching%2E+IEEE+Trans%2E+Commun%2E%2C+1984%2E)
12. Josh Coalson. Free lossless audio codec, 2008. URL [https://xiph.org/flac](https://xiph.org/flac) . [↗](https://xiph.org/flac)
13. David Cox. Syntactically informed text compression with recurrent neural networks. arXiv:1608.02893, 2016. — [arXiv:1608.02893](https://arxiv.org/abs/1608.02893) [find](https://scholar.google.com/scholar?q=David+Cox%2E+Syntactically+informed+text+compression+with+recurrent+neural+networks%2E+arXiv%3A1608%2E02893%2C+2016%2E+%E2%80%94+%5BarXiv%3A1608%2E02893%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1608%2E02893%29)
14. Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In ICLR, 2023. [our card](https://md.research.elamur.ai/free/en/papers/deletang-2022-neural-networks-chomsky-hierarchy/)
15. Peter Deutsch. GZIP file format specification version 4.3. RFC, 1996. [find](https://scholar.google.com/scholar?q=Peter+Deutsch%2E+GZIP+file+format+specification+version+4%2E3%2E+RFC%2C+1996%2E)
16. Jarek Duda. Asymmetric numeral systems. arXiv:0902.0271, 2009. — [arXiv:0902.0271](https://arxiv.org/abs/0902.0271) [find](https://scholar.google.com/scholar?q=Jarek+Duda%2E+Asymmetric+numeral+systems%2E+arXiv%3A0902%2E0271%2C+2009%2E+%E2%80%94+%5BarXiv%3A0902%2E0271%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F0902%2E0271%29)
17. Eibe Frank, Chang Chui, and Ian H. Witten. Text categorization using compression models. In Data Compression Conference, 2000. [find](https://scholar.google.com/scholar?q=Eibe+Frank%2C+Chang+Chui%2C+and+Ian+H%2E+Witten%2E+Text+categorization+using+compression+models%2E+In+Data+Compression+Conference%2C+2000%2E)
18. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv:2307.06945, 2023. [find](https://scholar.google.com/scholar?q=Tao+Ge%2C+Jing+Hu%2C+Xun+Wang%2C+Si%2DQing+Chen%2C+and+Furu+Wei%2E+In%2Dcontext+autoencoder+for+context+compression+in+a+large+language+model%2E+arXiv%3A2307%2E06945%2C+2023%2E)
19. Tim Genewein, Grégoire Delétang, Anian Ruoss, Li Kevin Wenliang, Elliot Catt, Vincent Dutordoir, Jordi Grau-Moya, Laurent Orseau, Marcus Hutter, and Joel Veness. Memory-based meta-learning on non-stationary distributions. In ICML, 2023. [find](https://scholar.google.com/scholar?q=Tim+Genewein%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Anian+Ruoss%2C+Li+Kevin+Wenliang%2C+Elliot+Catt%2C+Vincent+Dutordoir%2C+Jordi+Grau%2DMoya%2C+Laurent+Orseau%2C+Marcus+Hutter%2C+and+Joel+Veness%2E+Memory%2Dbased+meta%2Dlearning+on+non%2Dstat)
20. Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Deepzip: Lossless data compression using recurrent neural networks. In DCC, 2019. [find](https://scholar.google.com/scholar?q=Mohit+Goyal%2C+Kedar+Tatwawadi%2C+Shubham+Chandak%2C+and+Idoia+Ochoa%2E+Deepzip%3A+Lossless+data+compression+using+recurrent+neural+networks%2E+In+DCC%2C+2019%2E)
21. Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Dzip: Improved general-purpose lossless compression based on novel neural network modeling. In DCC, 2020. [find](https://scholar.google.com/scholar?q=Mohit+Goyal%2C+Kedar+Tatwawadi%2C+Shubham+Chandak%2C+and+Idoia+Ochoa%2E+Dzip%3A+Improved+general%2Dpurpose+lossless+compression+based+on+novel+neural+network+modeling%2E+In+DCC%2C+2020%2E)
22. Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. In NAACL-HLT (Findings), 2022. [our card](https://md.research.elamur.ai/free/en/papers/guo-2022-longt5-efficient-text-text-transformer/)
23. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. Training compute-optimal large language models. arXiv:2203.15556, 2022. [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
24. Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer discrete flows and lossless compression. In NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Jorn+W%2E+T%2E+Peters%2C+Rianne+van+den+Berg%2C+and+Max+Welling%2E+Integer+discrete+flows+and+lossless+compression%2E+In+NeurIPS%2C+2019%2E)
25. Paul G. Howard and Jeffrey Scott Vitter. Analysis of arithmetic coding for data compression. In Data Compression Conference, 1991. [find](https://scholar.google.com/scholar?q=Paul+G%2E+Howard+and+Jeffrey+Scott+Vitter%2E+Analysis+of+arithmetic+coding+for+data+compression%2E+In+Data+Compression+Conference%2C+1991%2E)
26. David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 1952. [find](https://scholar.google.com/scholar?q=David+A%2E+Huffman%2E+A+method+for+the+construction+of+minimum%2Dredundancy+codes%2E+Proceedings+of+the+IRE%2C+1952%2E)
27. Marcus Hutter. Universal Artificial Intellegence - Sequential Decisions Based on Algorithmic Probability. Springer, 2005. [find](https://scholar.google.com/scholar?q=Marcus+Hutter%2E+Universal+Artificial+Intellegence+%2D+Sequential+Decisions+Based+on+Algorithmic+Probability%2E+Springer%2C+2005%2E)
28. Marcus Hutter. 500’000C prize for compressing human knowledge, 2006. URL [http://prize.hutter1.net](http://prize.hutter1.net) . [↗](http://prize.hutter1.net)
29. Zhiying Jiang, Yiqin Dai, Ji Xin, Ming Li, and Jimmy Lin. Few-shot non-parametric learning with deep latent variable model. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Zhiying+Jiang%2C+Yiqin+Dai%2C+Ji+Xin%2C+Ming+Li%2C+and+Jimmy+Lin%2E+Few%2Dshot+non%2Dparametric+learning+with+deep+latent+variable+model%2E+In+NeurIPS%2C+2022%2E)
30. Zhiying Jiang, Matthew Y. R. Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin. "low-resource" text classification: A parameter-free classification method with compressors. In ACL (Findings), 2023. [find](https://scholar.google.com/scholar?q=Zhiying+Jiang%2C+Matthew+Y%2E+R%2E+Yang%2C+Mikhail+Tsirlin%2C+Raphael+Tang%2C+Yiqin+Dai%2C+and+Jimmy+Lin%2E+%22low%2Dresource%22+text+classification%3A+A+parameter%2Dfree+classification+method+with+compressors%2E+In+ACL+%28Finding)
31. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
32. Friso H. Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. In ICML, 2019. [find](https://scholar.google.com/scholar?q=Friso+H%2E+Kingma%2C+Pieter+Abbeel%2C+and+Jonathan+Ho%2E+Bit%2Dswap%3A+Recursive+bits%2Dback+coding+for+lossless+compression+with+hierarchical+latent+variables%2E+In+ICML%2C+2019%2E)
33. Byron Knoll. CMIX, 2014. URL [http://www.byronknoll.com/cmix.html](http://www.byronknoll.com/cmix.html) . [↗](http://www.byronknoll.com/cmix.html)
34. Byron Knoll and Nando de Freitas. A machine learning perspective on predictive coding with PAQ8. In DCC, 2012. [find](https://scholar.google.com/scholar?q=Byron+Knoll+and+Nando+de+Freitas%2E+A+machine+learning+perspective+on+predictive+coding+with+PAQ8%2E+In+DCC%2C+2012%2E)
35. Andrei N. Kolmogorov. On tables of random numbers. Theoretical Computer Science, 1998. [find](https://scholar.google.com/scholar?q=Andrei+N%2E+Kolmogorov%2E+On+tables+of+random+numbers%2E+Theoretical+Computer+Science%2C+1998%2E)
36. Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL (1), 2018. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
37. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP (Demonstration), 2018. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
38. Michael Laskin, Luyu Wang, et al. In-context reinforcement learning with algorithm distillation. In ICLR. OpenReview.net, 2023. [find](https://scholar.google.com/scholar?q=Michael+Laskin%2C+Luyu+Wang%2C+et+al%2E+In%2Dcontext+reinforcement+learning+with+algorithm+distillation%2E+In+ICLR%2E+OpenReview%2Enet%2C+2023%2E)
39. Ming Li and Paul M. B. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications, 4th Edition. Springer, 2019. [find](https://scholar.google.com/scholar?q=Ming+Li+and+Paul+M%2E+B%2E+Vit%C3%A1nyi%2E+An+Introduction+to+Kolmogorov+Complexity+and+Its+Applications%2C+4th+Edition%2E+Springer%2C+2019%2E)
40. Qian Liu, Yiling Xu, and Zhu Li. DecMac: A deep context model for high efficiency arithmetic coding. In ICAIIC, 2019. [find](https://scholar.google.com/scholar?q=Qian+Liu%2C+Yiling+Xu%2C+and+Zhu+Li%2E+DecMac%3A+A+deep+context+model+for+high+efficiency+arithmetic+coding%2E+In+ICAIIC%2C+2019%2E)
41. David J. C. MacKay. Information theory, inference, and learning algorithms. Cambridge University Press, 2003. [find](https://scholar.google.com/scholar?q=David+J%2E+C%2E+MacKay%2E+Information+theory%2C+inference%2C+and+learning+algorithms%2E+Cambridge+University+Press%2C+2003%2E)
42. Matthew V. Mahoney. Fast text compression with neural networks. In FLAIRS, 2000. [find](https://scholar.google.com/scholar?q=Matthew+V%2E+Mahoney%2E+Fast+text+compression+with+neural+networks%2E+In+FLAIRS%2C+2000%2E)
43. Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue. TRACE: A fast transformer-based general-purpose lossless compressor. In WWW, 2022. [find](https://scholar.google.com/scholar?q=Yu+Mao%2C+Yufei+Cui%2C+Tei%2DWei+Kuo%2C+and+Chun+Jason+Xue%2E+TRACE%3A+A+fast+transformer%2Dbased+general%2Dpurpose+lossless+compressor%2E+In+WWW%2C+2022%2E)
44. Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Fabian+Mentzer%2C+Eirikur+Agustsson%2C+Michael+Tschannen%2C+Radu+Timofte%2C+and+Luc+Van+Gool%2E+Practical+full+resolution+learned+lossless+image+compression%2E+In+CVPR%2C+2019%2E)
45. Fabian Mentzer, Luc Van Gool, and Michael Tschannen. Learning better lossless compression using lossy compression. In CVPR, 2020. [find](https://scholar.google.com/scholar?q=Fabian+Mentzer%2C+Luc+Van+Gool%2C+and+Michael+Tschannen%2E+Learning+better+lossless+compression+using+lossy+compression%2E+In+CVPR%2C+2020%2E)
46. Tomas Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis, Brno Universtiy of Technology, 2012. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2E+Statistical+Language+Models+Based+on+Neural+Networks%2E+PhD+thesis%2C+Brno+Universtiy+of+Technology%2C+2012%2E)
47. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv:2307.04721, 2023. [find](https://scholar.google.com/scholar?q=Suvir+Mirchandani%2C+Fei+Xia%2C+Pete+Florence%2C+Brian+Ichter%2C+Danny+Driess%2C+Montserrat+Gonzalez+Arenas%2C+Kanishka+Rao%2C+Dorsa+Sadigh%2C+and+Andy+Zeng%2E+Large+language+models+as+general+pattern+machines%2E+arXiv%3A2)
48. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In NIPS, 2017. [find](https://scholar.google.com/scholar?q=Behnam+Neyshabur%2C+Srinadh+Bhojanapalli%2C+David+McAllester%2C+and+Nati+Srebro%2E+Exploring+generalization+in+deep+learning%2E+In+NIPS%2C+2017%2E)
49. Juri Opitz. Gzip versus bag-of-words for text classification, 2023. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2E+Gzip+versus+bag%2Dof%2Dwords+for+text+classification%2C+2023%2E)
50. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, 2015. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+Librispeech%3A+An+ASR+corpus+based+on+public+domain+audio+books%2E+In+ICASSP%2C+2015%2E)
51. Richard C. Pasco. Source coding algorithms for fast data compression (ph.d. thesis abstr.). IEEE Trans. Inf. Theory, 1977. [find](https://scholar.google.com/scholar?q=Richard+C%2E+Pasco%2E+Source+coding+algorithms+for+fast+data+compression+%28ph%2Ed%2E+thesis+abstr%2E%29%2E+IEEE+Trans%2E+Inf%2E+Theory%2C+1977%2E)
52. Igor Pavlov. 7z Format, 2019. URL [http://www.7-zip.org/7z.html](http://www.7-zip.org/7z.html) . [↗](http://www.7-zip.org/7z.html)
53. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. In ACL, 2020. [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
54. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2C+2019%2E)
55. Jack W. Rae et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae+et+al%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+gopher%2E+arXiv%3A2112%2E11446%2C+2021%2E)
56. Samuel Rathmanner and Marcus Hutter. A philosophical treatise of universal induction. Entropy, 2011. [find](https://scholar.google.com/scholar?q=Samuel+Rathmanner+and+Marcus+Hutter%2E+A+philosophical+treatise+of+universal+induction%2E+Entropy%2C+2011%2E)
57. Hochang Rhee, Yeong Il Jang, Seyun Kim, and Nam Ik Cho. LC-FDNet: Learned lossless image compression with frequency decomposition network. In CVPR, 2022. [find](https://scholar.google.com/scholar?q=Hochang+Rhee%2C+Yeong+Il+Jang%2C+Seyun+Kim%2C+and+Nam+Ik+Cho%2E+LC%2DFDNet%3A+Learned+lossless+image+compression+with+frequency+decomposition+network%2E+In+CVPR%2C+2022%2E)
58. Jorma Rissanen. Generalized kraft inequality and arithmetic coding. IBM J. Res. Dev., 1976. [find](https://scholar.google.com/scholar?q=Jorma+Rissanen%2E+Generalized+kraft+inequality+and+arithmetic+coding%2E+IBM+J%2E+Res%2E+Dev%2E%2C+1976%2E)
59. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. In ACL (2), 2023. [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+Randomized+positional+encodings+boost+length+generalization+of+transformers%2E+)
60. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 2015. [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
61. Ionut Schiopu and Adrian Munteanu. Deep-learning-based lossless image coding. IEEE Trans. Circuits Syst. Video Technol., 2020. [find](https://scholar.google.com/scholar?q=Ionut+Schiopu+and+Adrian+Munteanu%2E+Deep%2Dlearning%2Dbased+lossless+image+coding%2E+IEEE+Trans%2E+Circuits+Syst%2E+Video+Technol%2E%2C+2020%2E)
62. Ionut Schiopu, Yu Liu, and Adrian Munteanu. CNN-based prediction for lossless coding of photographic images. In PCS, 2018. [find](https://scholar.google.com/scholar?q=Ionut+Schiopu%2C+Yu+Liu%2C+and+Adrian+Munteanu%2E+CNN%2Dbased+prediction+for+lossless+coding+of+photographic+images%2E+In+PCS%2C+2018%2E)
63. Jürgen Schmidhuber and Stefan Heil. Predictive coding with neural nets: Application to text compression. In NIPS, pp. 1047–1054. MIT Press, 1994. [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Schmidhuber+and+Stefan+Heil%2E+Predictive+coding+with+neural+nets%3A+Application+to+text+compression%2E+In+NIPS%2C+pp%2E+1047%E2%80%931054%2E+MIT+Press%2C+1994%2E)
64. Jürgen Schmidhuber and Stefan Heil. Sequential neural text compression. IEEE Trans. Neural Networks, 1996. [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Schmidhuber+and+Stefan+Heil%2E+Sequential+neural+text+compression%2E+IEEE+Trans%2E+Neural+Networks%2C+1996%2E)
65. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1), 2016. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
66. Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., 1948. [find](https://scholar.google.com/scholar?q=Claude+E%2E+Shannon%2E+A+mathematical+theory+of+communication%2E+Bell+Syst%2E+Tech%2E+J%2E%2C+1948%2E)
67. Ray J. Solomonoff. A formal theory of inductive inference. part I. Inf. Control., 1964a. [find](https://scholar.google.com/scholar?q=Ray+J%2E+Solomonoff%2E+A+formal+theory+of+inductive+inference%2E+part+I%2E+Inf%2E+Control%2E%2C+1964a%2E)
68. Ray J. Solomonoff. A formal theory of inductive inference. part II. Inf. Control., 1964b. [find](https://scholar.google.com/scholar?q=Ray+J%2E+Solomonoff%2E+A+formal+theory+of+inductive+inference%2E+part+II%2E+Inf%2E+Control%2E%2C+1964b%2E)
69. Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In ACL (1), 2022. [find](https://scholar.google.com/scholar?q=Chaofan+Tao%2C+Lu+Hou%2C+Wei+Zhang%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Qun+Liu%2C+Ping+Luo%2C+and+Ngai+Wong%2E+Compression+of+generative+pre%2Dtrained+language+models+via+quantization%2E+In+ACL+%281%29%2C+2022%2E)
70. William J. Teahan and David J. Harper. Using Compression-Based Language Models for Text Categorization, pp. 141–165. Springer Netherlands, 2003. [find](https://scholar.google.com/scholar?q=William+J%2E+Teahan+and+David+J%2E+Harper%2E+Using+Compression%2DBased+Language+Models+for+Text+Categorization%2C+pp%2E+141%E2%80%93165%2E+Springer+Netherlands%2C+2003%2E)
71. Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
72. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
73. James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In ICLR (Poster), 2019. [find](https://scholar.google.com/scholar?q=James+Townsend%2C+Thomas+Bird%2C+and+David+Barber%2E+Practical+lossless+compression+with+latent+variables+using+bits+back+coding%2E+In+ICLR+%28Poster%29%2C+2019%2E)
74. Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-François Cham-berland, and Srinivas Shakkottai. Llmzip: Lossless text compression using large language models. arXiv:2306.04050, 2023. — [arXiv:2306.04050](https://arxiv.org/abs/2306.04050) [find](https://scholar.google.com/scholar?q=Chandra+Shekhara+Kaushik+Valmeekam%2C+Krishna+Narayanan%2C+Dileep+Kalathil%2C+Jean%2DFran%C3%A7ois+Cham%2Dberland%2C+and+Srinivas+Shakkottai%2E+Llmzip%3A+Lossless+text+compression+using+large+language+models%2E+arXiv%3A2306%2E0)
75. Aäron van den Oord and Benjamin Schrauwen. The student-t mixture as a natural image patch prior with application to image compression. J. Mach. Learn. Res., 2014. [find](https://scholar.google.com/scholar?q=A%C3%A4ron+van+den+Oord+and+Benjamin+Schrauwen%2E+The+student%2Dt+mixture+as+a+natural+image+patch+prior+with+application+to+image+compression%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+2014%2E)
76. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
77. Joel Veness, Marc G. Bellemare, Marcus Hutter, Alvin Chua, and Guillaume Desjardins. Compress and control. In AAAI, 2015. [find](https://scholar.google.com/scholar?q=Joel+Veness%2C+Marc+G%2E+Bellemare%2C+Marcus+Hutter%2C+Alvin+Chua%2C+and+Guillaume+Desjardins%2E+Compress+and+control%2E+In+AAAI%2C+2015%2E)
78. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
79. Terry A. Welch. A technique for high-performance data compression. Computer, 1984. [find](https://scholar.google.com/scholar?q=Terry+A%2E+Welch%2E+A+technique+for+high%2Dperformance+data+compression%2E+Computer%2C+1984%2E)
80. Frans M. J. Willems, Yuri M. Shtarkov, and Tjalling J. Tjalkens. The context-tree weighting method: basic properties. IEEE Trans. Inf. Theory, 1995. [find](https://scholar.google.com/scholar?q=Frans+M%2E+J%2E+Willems%2C+Yuri+M%2E+Shtarkov%2C+and+Tjalling+J%2E+Tjalkens%2E+The+context%2Dtree+weighting+method%3A+basic+properties%2E+IEEE+Trans%2E+Inf%2E+Theory%2C+1995%2E)
81. Ian H. Witten, Radford M. Neal, and John G. Cleary. Arithmetic coding for data compression. Commun. ACM, 1987. [find](https://scholar.google.com/scholar?q=Ian+H%2E+Witten%2C+Radford+M%2E+Neal%2C+and+John+G%2E+Cleary%2E+Arithmetic+coding+for+data+compression%2E+Commun%2E+ACM%2C+1987%2E)
82. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In NeurIPS, 2020. [our card](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)