A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference
2026-07-29 • Performance
Performance
AI summaryⓘ
The authors look at how big language models (LLMs) need huge amounts of fast memory, which is hard to provide with current technology. They found that using regular computer memory is faster than recalculating data but can only handle a small number of users at once. To fix this, they designed a new system called the Marvell Photonic Fabric Memory Appliance that uses light-based connections instead of traditional electrical ones, allowing many computers to share a large memory pool with less delay. Their tests show this new system can make LLMs respond faster and more smoothly during conversations.
LLM inferenceKV cachememory wallCXL (Compute Express Link)photonic interconnectcrossbar topologylatencyhost memorymulti-turn conversationscache eviction
Authors
Jing Ding, Yash Nishant, Chandrish Ambati, Jyothsna Kamati, Trung Diep
Abstract
LLM inference at scale faces a memory wall. The KV cache demands tens of terabytes at hundreds of gigabytes per second, yet no current memory tier delivers both at once. Characterization across multi-generation GPU systems with various LLaMA models shows host memory retrieval achieves up to 100x speedup over re-computation but supports only tens of concurrent long-context users. Electrical CXL pooling theoretically bridges this gap, but switch latency, cable reach limits, and power-scaling issues prevent practical TB-scale deployments. We present the Marvell Photonic Fabric Memory Appliance, a photonic-CXL hybrid architecture replacing electrical switches with a passive fiber shuffle to deliver 32 TB shared memory across 16 hosts via a switch-free full- crossbar topology. Emulation results demonstrate over 50 percent latency reduction versus electrical CXL pools. Simulation results show that the PF Memory Appliance eliminates cache eviction cliffs by improving time-to-first-token by 6.6x for multi-turn conversations workloads.