Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
2026-07-28 • Machine Learning
Machine Learning
AI summaryⓘ
The authors propose CARE, a method that adjusts how many experts from a model help with each word based on how uncertain the model is about that word. Instead of always using a fixed number of experts, CARE uses the model's own confidence signals to decide when to stop adding experts, saving effort on easy words and focusing more on hard ones. This method works without adding extra parameters and improves performance on various tasks while using fewer computational resources. CARE also helps detect when the model encounters unfamiliar input better than other methods.
Mixture-of-ExpertsLow-Rank Adaptation (LoRA)routertoken uncertaintyconfidence-adaptive routingnucleus samplingout-of-distribution detectionepistemic uncertaintymodel calibrationLLaMA
Authors
Tom Saliencro, Rohan Desai, Priya Nair, Maya Lindqvist, Daniel Whitmore
Abstract
Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts $k$. Tokens differ in how uncertain the model is about them, so a single k over-spends on easy tokens and under-serves hard ones. We observe that the router's output distribution is already a per-token uncertainty signal: peaked mass indicates confidence, while a flat distribution indicates ambiguity. We introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion. Experts are activated in decreasing router weight until their cumulative mass reaches a threshold, with a small extension when the admitted experts disagree. A budget thermostat calibrates the threshold so that the average number of active experts matches any target. CARE is a drop-in, single-forward-pass rule with no extra parameters. Across eight commonsense benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, as well as math, code, and knowledge tasks, CARE improves over fixed top-k MoE-LoRA at matched compute and matches the fixed-k=4 baseline while activating fewer experts. The same confidence and disagreement signals also improve out-of-distribution detection over MSP, entropy, and multi-pass proxies. We support the design with nucleus fidelity, budget optimality, and an epistemic reading of disagreement, and we release code.