04 — siblings

kv-cache/

decoding/quantisation/kernels/serving/

Inference / kv-cache

KV cache

Autoregressive decoding is not compute-bound; it is bound by moving cached keys and values. Shrinking that cache, sharing it across heads, paging it, or reusing it across requests are all the same optimisation approached from different sides.

3 entries, 1 of them a stub.

Entries

Multi-Query Attentionstale

Single shared K/V head across query heads to shrink the decode cache.

Paged KV Cachestubstandard

Virtual-memory paging applied to the attention cache.

Prefix Cachingstandard

Reuse KV blocks across requests that share a prompt prefix.