Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions
2026-03-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors review ways to combine multiple trained large language models into one without needing extra training, which saves time and computing power. They organize these methods using a framework called FUSE that looks at the basics, strategies, use cases, and available tools. Their survey covers the theory behind merging models, different technical approaches like averaging weights or using expert mixtures, and applications in tasks like multi-task learning and language transfer. They also point out current challenges such as understanding the theory better and handling bigger models. Overall, the authors provide a clear guide for researchers interested in merging models efficiently.
model merginglarge language modelsweight averagingtask vectorsmulti-task learningloss landscapemode connectivityfederated learningmixture-of-expertsfine-tuning
Authors
Mingyang Song, Mao Zheng
Abstract
Model merging has emerged as a transformative paradigm for combining the capabilities of multiple neural networks into a single unified model without additional training. With the rapid proliferation of fine-tuned large language models~(LLMs), merging techniques offer a computationally efficient alternative to ensembles and full retraining, enabling practitioners to compose specialized capabilities at minimal cost. This survey presents a comprehensive and structured examination of model merging in the LLM era through the \textbf{FUSE} taxonomy, a four-dimensional framework organized along \textbf{F}oundations, \textbf{U}nification Strategies, \textbf{S}cenarios, and \textbf{E}cosystem. We first establish the theoretical underpinnings of merging, including loss landscape geometry, mode connectivity, and the linear mode connectivity hypothesis. We then systematically review the algorithmic landscape, spanning weight averaging, task vector arithmetic, sparsification-enhanced methods, mixture-of-experts architectures, and evolutionary optimization approaches. For each method family, we analyze the core formulation, highlight representative works, and discuss practical trade-offs. We further examine downstream applications across multi-task learning, safety alignment, domain specialization, multilingual transfer, and federated learning. Finally, we survey the supporting ecosystem of open-source tools, community platforms, and evaluation benchmarks, and identify key open challenges including theoretical gaps, scalability barriers, and standardization needs. This survey aims to equip researchers and practitioners with a structured foundation for advancing model merging.