BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
2026-07-09 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a problem in running many diffusion large language model (dLLM) requests together, where some requests finish faster than others, causing delays. They introduce BlockServe, a system that breaks work into blocks and immediately finishes shorter requests to keep things moving smoothly. Their method also smartly manages different request types and refills work efficiently to handle more requests at once. Testing showed BlockServe is much faster than previous methods while maintaining similar result quality.
diffusion large language modelsbatchingblock-grained schedulingheterogeneous convergenceparallel decodingcache managementinference throughputtoken-budgetingtail latencycompute bubbles
Authors
Yuanjie Zhu, Liangwei Yang, Ke Xu, Weizhi Zhang, Shanghao Li, Zihe Song, Philip S. Yu
Abstract
Efficient serving of diffusion large language models (dLLMs) is hindered by convergence heterogeneity: when batching multiple requests, different sequences converge at different rates, causing faster requests to stall behind slower stragglers and introducing compute bubbles and tail latency. We present BlockServe, a continuous batching framework that integrates block-grained scheduling -- immediately evicting completed requests at block boundaries -- with mixed-state execution that extends dual cache and parallel decoding to heterogeneous batches via gather-scatter indexing. Furthermore, a compute-aware admission controller expands effective batch capacity through token-budgeted refill. On Dream and LLaDA across five benchmarks, BlockServe achieves 1.9--10.6$\times$ throughput over Fast-dLLM with comparable generation quality, establishing block-grained scheduling as a foundation for high-throughput offline dLLM inference.