---
title: "A Label is Worth a Thousand Images in Dataset Distillation (2024)"
description: "Dataset distillation compresses a large training set into a small synthetic one that… 37.8% accuracy. 18 references, cited by 7 papers here."
authors: ["Tian Qin", "Zhiwei Deng", "David Alvarez-Melis"]
year: 2024
venue: "arXiv:2406.10485"
source: "https://arxiv.org/abs/2406.10485"
doi: "10.48550/arxiv.2406.10485"
arxiv: "2406.10485"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qin-2024-label-worth-thousand-images-dataset/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qin-2024-label-worth-thousand-images-dataset/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/) / A Label is Worth a Thousand Images in Dataset Distillation

# A Label is Worth a Thousand Images in Dataset Distillation

**Authors:** Tian Qin, Zhiwei Deng and [David Alvarez-Melis](https://md.research.elamur.ai/free/en/authors/david-alvarez-melis/) · **Year:** 2024 · **Venue:** [arXiv:2406.10485](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.10485) · [PDF](https://arxiv.org/pdf/2406.10485) · [DOI 10.48550/arxiv.2406.10485](https://doi.org/10.48550/arxiv.2406.10485) · [arXiv 2406.10485](https://arxiv.org/abs/2406.10485) · [OpenAlex](https://openalex.org/W4415800763)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 7 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Dataset distillation compresses a large training set into a small synthetic one that trains models almost as well, but leading methods generate synthetic images in very different ways, so it is unclear what drives their success. The authors observe that nearly all methods that scale to ImageNet-1K share one feature: soft (probabilistic) labels, often produced by pretrained expert models. They run controlled ablations on ImageNet-1K and TinyImageNet, including a simple baseline that labels randomly sampled real training images with expert-generated soft labels. This baseline performs on par with SOTA methods such as SRe2L, which shows that label quality, not image synthesis, explains most of the gains. Soft labels help only when they carry semantic structure, such as similarity between related classes. The best labels come from experts stopped early, and the ideal stopping point depends on the images-per-class budget. The paper also fits a data-knowledge scaling law and empirically maps the Pareto frontier for learning efficiently from limited data. Together, these results suggest that distillation research should focus more on labels.

## Key points
- On ImageNet-1K with random real images, hard labels give only 37.8% accuracy at IPC=200. Soft-label baselines using ResNet18 or ResNet50 experts reach about 57–63%. For comparison, full-data training yields 69.8% for ResNet18 and 76.1% for ResNet50.
- Soft labels must contain structured inter-class information to help. Different distillation budgets favor different label "profiles," which can be tuned by early-stopping the expert labeler.
- Ensembles of experts further improve the learning value of soft labels.
- Learning labels alone with truncated BPTT on TinyImageNet, without training any expert, recovers labels close to those of early-stopped experts. At IPC=1 the closest match is at epoch 13, near the optimal stop at epoch 11. In this label-only setting, dataset distillation converges to knowledge distillation.
- Code is available at github.com/sunnytqin/no-distillation. The experiments cover only image classification.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: Tian Qin, Zhiwei Deng and David Alvarez-Melis. 2024. A Label is Worth a Thousand Images in Dataset Distillation. arXiv:2406.10485.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qin2024label,
  title = {{A Label is Worth a Thousand Images in Dataset Distillation}},
  author = {Tian Qin, Zhiwei Deng and David Alvarez-Melis},
  year = {2024},
  doi = {10.48550/arxiv.2406.10485},
  eprint = {2406.10485},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.10485}
}
```

- [← previous A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions](https://md.research.elamur.ai/free/en/papers/resende-2024-comprehensive-view-biases-toxicity-sentiment/)
- [next → AlexUNLP-BH at StanceEval2024: Multiple Contrastive Losses Ensemble Strategy with Multi-Task Learning For Stance Detection in Arabic](https://md.research.elamur.ai/free/en/papers/badran-2024-alexunlp-bh-stanceeval2024-multiple-contrastive/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/)

## References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abdin et al. “[Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)”. arXiv [cs.CL] (2024). arXiv: 2404.14219 [cs.CL]. J. Ba and R. Caruana. “Do deep nets really need to be deep?” Adv. Neural Inf. Process. Syst. (2013), pp. 2654–2662. eprint: 1312.6184. O. Bohdal, Y. Yang,…
2. N. Sachdeva and J. McAuley. “[Data Distillation: A Survey](https://md.research.elamur.ai/free/en/papers/sachdeva-2023-data-distillation-survey/)”. arXiv [cs.LG] (2023). arXiv: 2301.04272 [cs.LG].
3. S. Shao et al. “Generalized large-scale data condensation via various backbone and Statistical Matching”. arXiv [cs.CV] (2023). arXiv: 2311.17950 [cs.CV]. — [arXiv:2311.17950](https://arxiv.org/abs/2311.17950) [find](https://scholar.google.com/scholar?q=S%2E+Shao+et+al%2E+%E2%80%9CGeneralized+large%2Dscale+data+condensation+via+various+backbone+and+Statistical+Matching%E2%80%9D%2E+arXiv+%5Bcs%2ECV%5D+%282023%29%2E+arXiv%3A+2311%2E17950+%5Bcs%2ECV%5D%2E+%E2%80%94+%5BarXiv%3A2311%2E17950%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F23)
4. Z. Shen and E. Xing. “A Fast Knowledge Distillation Framework for Visual Recognition”. arXiv [cs.CV] (2021). arXiv: 2112.01528 [cs.CV]. — [arXiv:2112.01528](https://arxiv.org/abs/2112.01528) [find](https://scholar.google.com/scholar?q=Z%2E+Shen+and+E%2E+Xing%2E+%E2%80%9CA+Fast+Knowledge+Distillation+Framework+for+Visual+Recognition%E2%80%9D%2E+arXiv+%5Bcs%2ECV%5D+%282021%29%2E+arXiv%3A+2112%2E01528+%5Bcs%2ECV%5D%2E+%E2%80%94+%5BarXiv%3A2112%2E01528%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2112%2E01528%29)
5. I. Sucholutsky and M. Schonlau. “Soft-Label Dataset Distillation and Text Dataset Distillation”. 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021, pp. 1–8. [find](https://scholar.google.com/scholar?q=I%2E+Sucholutsky+and+M%2E+Schonlau%2E+%E2%80%9CSoft%2DLabel+Dataset+Distillation+and+Text+Dataset+Distillation%E2%80%9D%2E+2021+International+Joint+Conference+on+Neural+Networks+%28IJCNN%29%2E+IEEE%2C+2021%2C+pp%2E+1%E2%80%938%2E)
6. Y. Tian, D. Krishnan, and P. Isola. “Contrastive Representation Distillation”. arXiv [cs.LG] (2019). arXiv: 1910.10699 [cs.LG]. — [arXiv:1910.10699](https://arxiv.org/abs/1910.10699) [find](https://scholar.google.com/scholar?q=Y%2E+Tian%2C+D%2E+Krishnan%2C+and+P%2E+Isola%2E+%E2%80%9CContrastive+Representation+Distillation%E2%80%9D%2E+arXiv+%5Bcs%2ELG%5D+%282019%29%2E+arXiv%3A+1910%2E10699+%5Bcs%2ELG%5D%2E+%E2%80%94+%5BarXiv%3A1910%2E10699%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E10699%29)
7. K. Wang et al. “CAFE: Learning to Condense dataset by Aligning FEatures”. arXiv [cs.CV] (2022), pp. 12196–12205. arXiv: 2203.01531 [cs.CV]. — [arXiv:2203.01531](https://arxiv.org/abs/2203.01531) [find](https://scholar.google.com/scholar?q=K%2E+Wang+et+al%2E+%E2%80%9CCAFE%3A+Learning+to+Condense+dataset+by+Aligning+FEatures%E2%80%9D%2E+arXiv+%5Bcs%2ECV%5D+%282022%29%2C+pp%2E+12196%E2%80%9312205%2E+arXiv%3A+2203%2E01531+%5Bcs%2ECV%5D%2E+%E2%80%94+%5BarXiv%3A2203%2E01531%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2203%2E01531%29)
8. T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. “[Dataset Distillation](https://md.research.elamur.ai/free/en/papers/wang-2018-dataset-distillation/)”. arXiv [cs.LG] (2018). arXiv: 1811.10959 [cs.LG].
9. W. Yang, Y. Zhu, Z. Deng, and O. Russakovsky. “What is Dataset Distillation Learning?” arXiv [cs.LG] (2024). arXiv: 2406.04284 [cs.LG]. [find](https://scholar.google.com/scholar?q=W%2E+Yang%2C+Y%2E+Zhu%2C+Z%2E+Deng%2C+and+O%2E+Russakovsky%2E+%E2%80%9CWhat+is+Dataset+Distillation+Learning%3F%E2%80%9D+arXiv+%5Bcs%2ELG%5D+%282024%29%2E+arXiv%3A+2406%2E04284+%5Bcs%2ELG%5D%2E)
10. H. Yin et al. “Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion”. arXiv [cs.LG] (2019). arXiv: 1912.08795 [cs.LG]. — [arXiv:1912.08795](https://arxiv.org/abs/1912.08795) [find](https://scholar.google.com/scholar?q=H%2E+Yin+et+al%2E+%E2%80%9CDreaming+to+Distill%3A+Data%2Dfree+Knowledge+Transfer+via+DeepInversion%E2%80%9D%2E+arXiv+%5Bcs%2ELG%5D+%282019%29%2E+arXiv%3A+1912%2E08795+%5Bcs%2ELG%5D%2E+%E2%80%94+%5BarXiv%3A1912%2E08795%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E08795%29)
11. Z. Yin and Z. Shen. “Dataset Distillation in Large Data Era”. arXiv [cs.CV] (2023). arXiv: 2311.18838 [cs.CV]. — [arXiv:2311.18838](https://arxiv.org/abs/2311.18838) [find](https://scholar.google.com/scholar?q=Z%2E+Yin+and+Z%2E+Shen%2E+%E2%80%9CDataset+Distillation+in+Large+Data+Era%E2%80%9D%2E+arXiv+%5Bcs%2ECV%5D+%282023%29%2E+arXiv%3A+2311%2E18838+%5Bcs%2ECV%5D%2E+%E2%80%94+%5BarXiv%3A2311%2E18838%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E18838%29)
12. Z. Yin, E. Xing, and Z. Shen. “Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective”. arXiv [cs.CV] (2023). arXiv: 2306.13092 [cs.CV]. — [arXiv:2306.13092](https://arxiv.org/abs/2306.13092) [find](https://scholar.google.com/scholar?q=Z%2E+Yin%2C+E%2E+Xing%2C+and+Z%2E+Shen%2E+%E2%80%9CSqueeze%2C+Recover+and+Relabel%3A+Dataset+Condensation+at+ImageNet+Scale+From+A+New+Perspective%E2%80%9D%2E+arXiv+%5Bcs%2ECV%5D+%282023%29%2E+arXiv%3A+2306%2E13092+%5Bcs%2ECV%5D%2E+%E2%80%94+%5BarXiv%3A2306%2E13092%5D%28https)
13. L. Yuan et al. “Revisiting Knowledge Distillation via label smoothing regularization”. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2020), pp. 3902–3910. [find](https://scholar.google.com/scholar?q=L%2E+Yuan+et+al%2E+%E2%80%9CRevisiting+Knowledge+Distillation+via+label+smoothing+regularization%E2%80%9D%2E+Proc%2E+IEEE+Comput%2E+Soc%2E+Conf%2E+Comput%2E+Vis%2E+Pattern+Recognit%2E+%282020%29%2C+pp%2E+3902%E2%80%933910%2E)
14. B. Zhao and H. Bilen. “Dataset Condensation with Differentiable Siamese Augmentation”. Proceedings of the 38th International Conference on Machine Learning. Ed. by M. Meila and T. Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 12674–12685. [find](https://scholar.google.com/scholar?q=B%2E+Zhao+and+H%2E+Bilen%2E+%E2%80%9CDataset+Condensation+with+Differentiable+Siamese+Augmentation%E2%80%9D%2E+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2E+Ed%2E+by+M%2E+Meila+and+T%2E+Zhang%2E+Vol%2E+139%2E+Pro)
15. B. Zhao and H. Bilen. “Dataset Condensation with Distribution Matching”. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023, pp. 6514–6523. [find](https://scholar.google.com/scholar?q=B%2E+Zhao+and+H%2E+Bilen%2E+%E2%80%9CDataset+Condensation+with+Distribution+Matching%E2%80%9D%2E+2023+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision+%28WACV%29%2E+IEEE%2C+2023%2C+pp%2E+6514%E2%80%936523%2E)
16. B. Zhao, K. R. Mopuri, and H. Bilen. “[Dataset Condensation with Gradient Matching](https://md.research.elamur.ai/free/en/papers/zhao-2020-dataset-condensation-gradient-matching/)”. arXiv [cs.CV] (2020). arXiv: 2006.05929 [cs.CV].
17. H. Zhou et al. “Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective”. arXiv [cs.LG] (2021). arXiv: 2102.00650 [cs.LG]. — [arXiv:2102.00650](https://arxiv.org/abs/2102.00650) [find](https://scholar.google.com/scholar?q=H%2E+Zhou+et+al%2E+%E2%80%9CRethinking+Soft+Labels+for+Knowledge+Distillation%3A+A+Bias%2DVariance+Tradeoff+Perspective%E2%80%9D%2E+arXiv+%5Bcs%2ELG%5D+%282021%29%2E+arXiv%3A+2102%2E00650+%5Bcs%2ELG%5D%2E+%E2%80%94+%5BarXiv%3A2102%2E00650%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F21)
18. Y. Zhou, E. Nezhadarya, and J. Ba. “Dataset Distillation using Neural Feature Regression”. arXiv [cs.LG] (2022). arXiv: 2206.00719 [cs.LG]. — [arXiv:2206.00719](https://arxiv.org/abs/2206.00719) [find](https://scholar.google.com/scholar?q=Y%2E+Zhou%2C+E%2E+Nezhadarya%2C+and+J%2E+Ba%2E+%E2%80%9CDataset+Distillation+using+Neural+Feature+Regression%E2%80%9D%2E+arXiv+%5Bcs%2ELG%5D+%282022%29%2E+arXiv%3A+2206%2E00719+%5Bcs%2ELG%5D%2E+%E2%80%94+%5BarXiv%3A2206%2E00719%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E00719%29)

*Updated 3 November 2026*