E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
2026-06-02 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
AI summaryⓘ
The authors present E2LLM, a system to run Large Language Models (LLMs) efficiently on devices with limited resources, like those at the Edge or Fog. Instead of splitting one model across all devices, they replicate full models across groups and split the work inside each group based on different tasks during language processing. They use smart algorithms to group devices and decide the best way to divide the model. Their tests show that E2LLM can handle changing workloads well and significantly cuts down waiting times compared to existing methods.
Large Language ModelsEdge ComputingFog ComputingModel ParallelismGenetic AlgorithmDynamic ProgrammingToken ProcessingResource ConstraintsLatencyWorkload Adaptation
Authors
Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La, Frank Eliassen, Phuong Hoai Ha, Peiyuan Guan
Abstract
Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging. Beyond executing the models themselves, practical deployment must address cost efficiency, low latency, and optimal resource utilization. Conventional approaches typically assume that an entire model can be hosted on a single device, which does not hold in many real-world scenarios, particularly in Edge and Fog environments where device resources are constrained. In this paper, we introduce E2LLM, a framework designed to enable efficient LLM deployment in such resource limited settings. Rather than simply partitioning a single model across all available devices, E2LLM replicates the full model across multiple groups of devices (replicas) and applies model parallelism within each replica. Each replica is assigned a specialized role PREFILL or DECODER based on its efficiency in handling input and output tokens. This separation leverages the inherent differences between these two phases of LLM inference. To effectively organize devices, we utilize a Genetic Algorithm to form clusters that maximize system performance. Within each cluster, we apply Dynamic Programming to determine an optimal partitioning strategy that minimizes bottlenecks in model-parallel execution. Experimental results demonstrate that our approach adapts robustly to varying workloads, including scenarios with significant variation in input and output token lengths. Compared to the Splitwise baseline, E2LLM reduces average waiting time by over 50% under high-demand conditions