It pays off, but only under a condition most teams skip past: the sharing has to be measured in tokens saved per second of prefill, not in hit rate. Two numbers make that concrete.
First, what a token costs. KV bytes per token = 2 (K and V) × layers × KV heads × head dim × dtype bytes. For a Llama-3-8B shape, that is 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token. For a 70B with 80 layers and 8 KV heads, 2 × 80 × 8 × 128 × 2 = 320 KiB per token. A 4,096-token system prompt therefore pins 512 MiB on the 8B and 1.25 GiB on the 70B, per request, if nothing is shared. Thirty-two concurrent requests on the same prompt: 16 GiB versus 512 MiB plus suffixes. That is the payoff — it is a capacity multiplier, not a latency trick.
Second, what the saved prefill is worth. If 4k tokens prefill at a rate that takes 180 ms, a full prefix hit removes those 180 ms; comparing that to a 30-token decode at 25 ms per token (750 ms), TTFT drops roughly 19%. On shorter prompts the percentage collapses: a 512-token prefix saves about 22 ms against the same 750 ms decode, which is inside noise. That is the real threshold — sharing is worth building when the shared prefix is long relative to the completion, which is exactly the RAG-with-long-instructions and agent-with-tool-schema case, not the chatbot case.
Now the part that decides whether it stays paid off.
Block granularity. Hash prompts in blocks — 16 tokens is the common choice — and key the cache on (model id, token ids, position). Two prompts that agree for 4,000 tokens but differ at token 33 share exactly two blocks. That single divisor is why "same system prompt" often yields nothing: a chat template that injects a session id or a timestamp at token 30 breaks the chain for everything downstream. Log the first divergent token index per request; if the distribution piles up at token 8–40, your template is the bug, not the cache.
Memory pressure versus hit rate. A radix-tree or LRU prefix store that is smaller than the working set thrashes: cache 60 GiB, working set 200 GiB, hit rate falls off a cliff and the gather cost is paid on every miss. Watch the eviction counter alongside the hit rate; a rising eviction count with flat hit rate means you are at capacity, not at a design limit.
Gather cost. Shared blocks are not contiguous in the paged KV layout, so attention has to gather across block tables. That is cheap when block tables are small and expensive when a request touches hundreds of extents. If TTFT gets worse with cache on, measure the copy time separately from the attention time before concluding the cache is useless.
Counting hit rate correctly. Request-level hit rate and token-level hit rate differ by a lot: 95% of requests hitting a 40-token prefix is roughly 30% of tokens. Report both, plus TTFT p50 and p99 with cache on and off, plus peak KV occupancy. If p99 TTFT regresses while p50 improves, you have a scheduler problem — cache hits arriving late in the batch, or prefetch blocking the decode step.
One design note on correctness: sharing must be token-exact, and position-dependent. Two requests with the same tokens but different positions after a shared prefix need the position offset applied at attention time; getting this wrong produces subtly wrong logits that no benchmark catches, because the first divergent token is where the error starts and it is usually a punctuation token.
Common misjudgment: assuming a hit means a fast request. A hit on 300 of 4,096 tokens is 7%, and the remaining 3,796 tokens still pay full prefill. The metric that matters is saved prefill milliseconds per second of GPU time, which is hit tokens × bytes/token ÷ prefill throughput, normalized by the memory you gave up.
Have you looked at the first-divergent-token distribution for your traffic? That one histogram usually explains more than the hit-rate dashboard, and it is a ten-line change to log.