Valkey Project Benchmarks a Shared Valkey KV Cache for LLM Inference, Reporting 25% More Requests Served on the Same Two GPU Nodes

Overview

A Valkey project blog post dated October 20, 2026, and written by Omer Rubinstein, Chaitanya Nuthalapati and Jeff Duffy, describes a benchmark in which a shared three-shard Valkey 9 cluster acted as an external cache for large language model (LLM) key-value (KV) data. According to the post, the same two GPU nodes served about 25% more requests with it than without it. The figures come from the authors’ own benchmark of the project’s own software, and the post says the traffic was synthetic.

What the Benchmark Measured

The authors say KV caching lets an inference engine store and reuse intermediate tensors for repeated context such as tool definitions, prompt instructions and conversation history. In their post, the setup was 100 concurrent conversations growing from 12K to 28K tokens of context, arriving at 8 requests per second across two GPU nodes with 8 H100 GPUs each, serving Llama-3.1-70B with vLLM plus LMCache. The only difference between the two test arms was whether a shared Valkey cache was attached as a second-level (L2) tier behind the GPU and host-memory tiers.

The post reports these results for a 20-minute run:

• Requests served: 3,692 without the Valkey tier and 4,631 with it, a 25% increase.
• Output tokens: about 28% more with the Valkey tier.
• Reply streaming speed: 11.9 tokens per second without it and 24.8 with it, which the post calls about 2.1 times faster.
• First token: 8.5 seconds without it and 0.13 seconds with it, for conversations under about 16K tokens of context while the fleet still had capacity headroom.
• Recompute share of prompt tokens: about 10% without it and about 1% with it.

On cost, the post says the three-shard cluster “cost about 15% on top of the GPU bill, so you pay 15% more to serve 25% more requests.”

Why a Shared Tier

The authors argue that GPU and host memory fill up because KV data is large. They put a single conversation of 20K to 30K tokens on Llama-3.1-70B at 6.5 to 10 GB of KV data, and say that returning users often land on a server that never saw their conversation. A shared Valkey tier, they write, lets any GPU node load any conversation’s KV, so load balancing, scale-out and restarts “stop costing you recomputes.”

The accompanying cookbook makes a related distinction, stating that vLLM’s own built-in prefix cache “only covers what a single engine holds and is not shared with other engines.” It describes LMCache, the layer that connects vLLM to a remote backend, and walks through a four-container stack of Valkey, a standalone LMCache server and two vLLM replicas. The cookbook says the whole thing runs on CPU, so it demonstrates cross-replica reuse rather than reproducing the H100 benchmark.

Limits the Authors Note

The post lists cases where the approach helps less. If a whole working set fits in GPU and CPU memory, an L2 tier is not needed yet. Low-reuse workloads with no growing history or shared prefix gain less. It also says a cache cannot create GPU capacity: at 8 requests per second on two nodes, conversations past about 16K tokens needed more prefill for their new tokens than the GPUs could deliver, so first-token latency stopped improving, although throughput and streaming speed still improved across the whole run.

The post also gives configuration notes: Valkey’s default noeviction policy “rejects writes once memory is full,” and the authors changed the eviction policy and capped each server’s host-memory tier at 20 GB to recreate the eviction pressure of a larger fleet.

What Is Not Established

The results are a single configuration of one model, one GPU type and synthetic traffic, reported by the project’s own contributors; the Herald found no independent replication.

One related item concerns LMCache itself. The Herald previously reported on a JFrog advisory about a critical flaw in LMCache’s multiprocess mode, with no fixed version listed as of October 7. The cookbook says its replicas hand KV to the standalone LMCache server “over the multi-process connector,” and its own security note states: “These examples use no authentication or TLS for simplicity.” Neither the post nor the cookbook mentions the advisory, and this article does not determine whether the cookbook’s configuration is affected.