AI summaryⓘ
The authors developed ATSInfer, a system that helps run large language models on everyday computers like laptops, where the GPU memory is often too small to hold the entire model. Unlike previous methods that move whole layers of the model between CPU and GPU, ATSInfer moves smaller pieces called tensors more smartly, adapting to how busy the hardware is. This approach uses a mix of planned and real-time decisions to efficiently share work between CPU and GPU. Their tests showed ATSInfer can run models nearly twice as fast in some tasks, making better use of the computer's resources. This means users can have a smoother experience running big language models right on their own devices.
Large Language ModelsGPU MemoryCPU-GPU OffloadingTensor GranularityHybrid ComputingModel InferenceLoad-aware SchedulingMoE ModelsPCIe BandwidthAsynchronous Coordination
Authors
Yangyijian Liu, Hongyi Ye, Mingyang Li, Wu-jun Li
Abstract
Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94$\times$ and decode throughput by up to 3.29$\times$, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.