Generating text means running the model again for every new token. Each run needs the keys (K) and values (V) of all earlier positions. A KV cache keeps them, so only the new position's Q, K and V are computed.
Prefill, then one token at a time
Toy calculation· operation counts are illustrativePrefill: the prompt is processed in one pass.
| position | 0: The | 1: ␣researcher | 2: ␣opened | 3: ␣the |
|---|---|---|---|---|
| K | new | new | new | new |
| V | new | new | new | new |
Legend (words, not only colour): new = computed this step; cached = read from memory; recomputed = computed again because nothing was kept. Queries are never cached: only the newest position needs one.
cache entries now
4
K/V computations so far (with cache)
4
K/V computations so far (without cache)
4
Counts are per layer and per KV head, for this toy prompt of 4 tokens. They illustrate how work grows; they are not timing measurements.
KV cache size for Qwen3-8B-Base
Recorded Qwen measurement- Captured
- computed from the verified configuration (layers, KV heads, head dimension)
- Shape
- K and V: [batch, 8 KV heads, tokens, 128] per layer, 36 layers
- Shown
- an estimate of the cache tensors only; excludes model weights, activations and allocator or framework overhead
dtype
Estimated KV cache: 576 MiB (bfloat16, batch 1, 4,096 tokens, overhead excluded)
With 32 KV heads (no grouping) it would be 2.25 GiB: 4 times more. Fewer KV heads is the point of grouped-query attention.