It Takes a MAESTRO To Prune Bad Experts
2026-07-09 • Computation and Language
Computation and Language
AI summaryⓘ
The authors introduce MAESTRO, a new way to shrink Mixture-of-Experts (MoE) language models by understanding how these models route information between different parts (experts) over time. Unlike older methods that look only at parts individually, MAESTRO treats the routing like a chain of connected decisions, helping to figure out which parts are most important overall. Tested on various tasks, their method keeps performance better while cutting the model size in half, and it works more reliably across different types of tasks. This helps make these large, efficient models easier to deploy in practice.
Mixture-of-Expertslanguage modelsstructured pruningautoregressiveMarkov chainsstationary distributionmodel compressioninference efficiencycross-layer dependenciesrouting
Authors
Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty
Abstract
Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.