Select to Think: Unlocking SLM Potential with Local Sufficiency
2026-04-29 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study small language models (SLMs), which are faster but less smart at reasoning than big language models (LLMs). They find that when the small model makes a different guess, the correct answer is often still among its top few choices. So, instead of having the big model generate answers, the authors let it just pick the best option from those the small model suggested. They teach the small model to do this choosing on its own, which improves its performance close to the big model without extra slow steps.
Small language models (SLMs)Large language models (LLMs)ReasoningToken predictionDistillationTop-K predictionsCandidate rankingGreedy decodingSelf-consistencyModel efficiency
Authors
Wenxuan Ye, Yangyang Zhang, Xueli An, Georg Carle, Yunpu Ma
Abstract
Small language models (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power exhibited by their larger counterparts (LLMs). To mitigate this gap, current approaches invoke an LLM to generate tokens at points of reasoning divergence, but these external calls introduce substantial latency and costs. Alternatively, standard distillation is often hindered by the capacity limitation, as SLMs struggle to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose SELECT TO THINK (S2T), which reframes the LLM's role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-LOCAL, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, we demonstrate that a 1.5B SLM's top-8 candidates capture the 32B LLM's choice with 95% hit rate. Translating this potential into performance, S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks, effectively matching the efficacy of 8-path self-consistency while operating with single-trajectory efficiency.