Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
2026-07-13 • Machine Learning
Machine Learning
AI summaryⓘ
The authors propose a new method called requential coding that compresses training data more efficiently by focusing only on the disagreements between a teacher and student model. Unlike previous methods, this technique produces shorter code lengths that do not depend on model size or data complexity. Their approach provides better generalization guarantees for large language models and helps explain why bigger models can learn more despite having more parameters. The method also distinguishes learnable patterns in data from random noise, showing that text data has more meaningful structure than images. Overall, the authors' work offers a new perspective on model compression and learning.
compressiongeneralizationlarge language modelsprequential codingrequential codingPAC-Bayes boundmodel sizeentropyparameter quantizationoverfitting
Authors
Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson
Abstract
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.