TTVS: Boosting Self-Exploring Reinforcement Learning via Test-time Variational Synthesis
2026-04-09 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a problem where large reasoning models struggle to learn in new or specialized areas without expensive supervision. They propose Test-Time Variational Synthesis (TTVS), a method that helps models improve themselves during testing by creating many different but similar versions of each test question. This encourages the models to understand the core idea rather than just memorizing patterns. Their experiments show that TTVS works well across different models and even beats other adaptation methods, despite only using unlabeled test data.
Large Reasoning Modelsreinforcement learningtest-time adaptationvariational synthesisself-supervised learningexploration-exploitationunlabeled datasemantic equivalence
Authors
Sikai Bai, Haoxi Li, Jie Zhang, Yongjiang Liu, Song Guo
Abstract
Despite significant advances in Large Reasoning Models (LRMs) driven by reinforcement learning with verifiable rewards (RLVR), this paradigm is fundamentally limited in specialized or novel domains where such supervision is prohibitively expensive or unavailable, posing a key challenge for test-time adaptation. While existing test-time methods offer a potential solution, they are constrained by learning from static query sets, risking overfitting to textual patterns. To address this gap, we introduce Test-Time Variational Synthesis (TTVS), a novel framework that enables LRMs to self-evolve by dynamically augmenting the training stream from unlabeled test queries. TTVS comprises two synergistic modules: (1) Online Variational Synthesis, which transforms static test queries into a dynamic stream of diverse, semantically-equivalent variations, enforcing the model to learn underlying problem logic rather than superficial patterns; (2) Test-time Hybrid Exploration, which balances accuracy-driven exploitation with consistency-driven exploration across synthetic variants. Extensive experiments show TTVS yields superior performance across eight model architectures. Notably, using only unlabeled test-time data, TTVS not only surpasses other test-time adaptation methods but also outperforms state-of-the-art supervised RL-based techniques trained on vast, high-quality labeled data.