NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
2026-06-02 • Performance
PerformanceArtificial IntelligenceDistributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
AI summaryⓘ
The authors study how big language models (LLMs) process data in a networked setup where parts of the model run on different machines. They find that current scheduling methods don't consider network delays well, which slows down the time before the first output token appears. To fix this, they introduce NetKV, which uses a new way to estimate network costs and schedule requests more efficiently. Their approach reduces waiting times and improves meeting performance goals without needing hardware changes.
Large Language ModelsKV cacheTime to First Token (TTFT)SchedulerCache localityNetwork congestionFat-tree topologyLoad balancingInference latencyDistributed computing
Authors
Mubarak Adetunji Ojewale
Abstract
Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we prove that ignoring the network term renders cache-aware-only scheduling arbitrarily suboptimal as context length grows. NetKV, the O(|D|) per-request greedy that consumes this oracle, has tier rankings that are provably robust to stale telemetry. On a 64-GPU four-tier fat-tree simulator driven by Mooncake traces, NetKV reduces mean TTFT by up to 21.2% over round-robin and 17.6% over a tuned cache+load-aware scheduler, lifts SLO attainment by up to 20.1 percentage points, and keeps the Time Between Tokens overhead below 0.5 ms in every condition tested, with no changes to the transport, inference engine, or hardware.