ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present ZO-Act, a new method for fine-tuning large language models without needing gradient calculations. Instead of changing all model weights or random directions, they limit changes to a small, smartly chosen space based on model activations, reducing noise and improving performance. This approach also works well with quantized models and common optimizers. Their experiments show ZO-Act outperforms existing methods on several large language models and tasks like question answering.
zeroth-order optimizationlarge language modelsactivation basislow-rank subspacequantized modelsAdam optimizerfinite-difference errormomentum-based optimizersLLM fine-tuningvariance reduction
Authors
Xun Dong, Yibo Xu, Naigang Wang, Xin Li, Penghang Yin, Zi Yang
Abstract
Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.