Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
2026-08-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study three ways to combine multiple specialized language models, called Merge, Mix RL, and MOPD, each reusing different parts of the experts. They tested all three methods on the same tasks and data, finding that overall performance is similar, but results vary by domain. Each method has unique strengths and restrictions, such as how they handle training data or expert knowledge. The authors suggest using Merge if experts already exist, Mix RL to train a new unified model, and MOPD for keeping domain-specific skills.
Reinforcement LearningLarge Language ModelsModel FusionDomain ExpertsTask VectorsDataset PoolingOn-Policy DistillationCross-Domain TransferTraining DynamicsPerformance Benchmarking
Authors
Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.