Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to efficiently represent large neural networks using smaller sets of parameters generated from a low-dimensional latent space. They focus on cases where a single generator creates the full network's parameters from a smaller vector and analyze the tradeoff between the size of this latent vector and the overall network size. Their main finding shows that even with a fixed-sized latent vector, the approximation error decreases as the network grows, following a precise mathematical rate. This work helps understand why and how methods like hypernetworks and parameter-efficient adaptations work well.
Neural networksParameter-efficient methodsLatent representationHypernetworksAffine generatorReLU architectureApproximation errorHölder functionsUniform approximationModel compression
Authors
Shijun Zhang
Abstract
Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture $Φ$ with $P_Φ$ parameter slots, we write $\boldsymbolθ_f=\mathcal{G}(\boldsymbolξ_f)$, where $\mathcal{G}\colon\mathbb{R}^M\to\mathbb{R}^{P_Φ}$ is a parameter generator and $\boldsymbolξ_f\in\mathbb{R}^M$ is a latent representation of the target function $f$. The architecture $Φ$ and the generator $\mathcal{G}$ are shared across the entire target class, while each target $f$ is represented by its own latent vector $\boldsymbolξ_f$, with $Φ_{\mathcal{G}(\boldsymbolξ_f)}$ approximating $f$. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension $M$ and the network budget $P$ is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures $Φ$ satisfying $P_Φ\leq P$ and affine generators $\mathcal{G}:\mathbb{R}^M\to \mathbb{R}^{P_Φ}$, we prove that the optimal worst-case uniform approximation error over the unit ball of $α$-Hölder functions on $[0,1]^d$, where $0<α\leq1$, has the sharp order $ \bigl(P\min\{M,P\}\bigr)^{-α/d}. $ In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.