StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration
2026-07-09 • Hardware Architecture
Hardware Architecture
AI summaryⓘ
The authors point out that making large language models smaller by compressing weights is common but slows down processing on GPUs because of how dequantization (converting compressed data back) is done. They introduce StreamDQ, a small hardware change that moves this dequantization step into the memory chips, making the process faster and more efficient. This change reduces work on the GPU, cuts energy use, and speeds up language model inference significantly. Their tests show StreamDQ can more than double the speed of decoding while saving a lot of power.
Large Language Models (LLMs)QuantizationDequantizationCUDA coresHigh-Bandwidth Memory (HBM)GEMM (General Matrix Multiply)InferenceHardware AccelerationMixed-Precision ComputingMemory Subsystem
Authors
Minki Jeong, Daegun Yoon, Soohong Ahn, Seungyong Lee, Nameun Kang, Hyeonseok Ju, Ieryung Park, Joonseop Sim, Youngpyo Joo, Hoshik Kim
Abstract
As large language models (LLMs) scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization to the memory side, StreamDQ eliminates GPU-side CUDA-core-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back and reload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08$\times$ speedup and 90.23\% lower energy for mixed-precision GEMM, with only 0.127\,mm$^2$ area and 0.355\,W power overhead per DQB in a 12\,nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68\% and improves decode throughput by up to 2.20$\times$.