Valkey Project Benchmarks a Shared Valkey KV Cache for LLM Inference, Reporting 25% More Requests Served on the Same Two GPU Nodes
A Valkey blog post reports that a three-shard Valkey 9 cache served 4,631 requests versus 3,692 on the same two 8-GPU nodes, at a cost of about 15% on top of the GPU bill.