Condensing Large-Scale Datasets Directly with Minimal Information Loss
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors found that current methods for shrinking large datasets into smaller synthetic ones lose important information because they compress data twice—from data to model and back to images—which makes label prediction less reliable. They showed this causes the common relabeling step to work poorly. To fix this, they created a new method called CIM that measures and minimizes the difference between the original and synthetic data directly, avoiding that harmful double compression. Their experiments show CIM works better and faster than previous methods, especially on big tasks like ImageNet. This helps create smaller, accurate datasets more efficiently.
dataset distillationsynthetic datainformation lossdual compressionrelabelingdistribution alignmentImageNetResNet-18top-1 accuracydata condensation
Authors
Xinyi Shang, Peng Sun, Bei Shi, Zixuan Wang, Tao Lin
Abstract
Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.